Efficient decisions for deblocking

ABSTRACT

The present invention relates to deblocking filtering, which may be advantageously applied for block-wise encoding and decoding of image or video signal. In particular, the present invention relates to performing an efficient and accurate decision on whether or not to apply deblocking filtering on an image block. The efficient and accurate decision is achieved by performing individual decisions on whether or not to apply deblocking filtering for segments of a boundary between adjacent image blocks, wherein the individual decision are based on pixels comprised in a subset of the pixel lines that the image blocks are composed of.

The present invention relates to the filtering of images. In particular,the present invention relates to deblocking filtering and to decisionson enabling or disabling deblocking filtering for an image block of avideo image.

BACKGROUND OF THE INVENTION

At present, the majority of standardized video coding algorithms arebased on hybrid video coding. Hybrid video coding methods typicallycombine several different lossless and lossy compression schemes inorder to achieve the desired compression gain. Hybrid video coding isalso the basis for ITU-T standards (H.26x standards such as H.261,H.263) as well as ISO/IEC standards (MPEG-X standards such as MPEG-1,MPEG-2, and MPEG-4). The most recent and advanced video coding standardis currently the standard denoted as H.264/MPEG-4 advanced video coding(AVC) which is a result of standardization efforts by joint video team(JVT), a joint team of ITU-T and ISO/IEC MPEG groups. This codec isbeing further developed by Joint Collaborative Team on Video Coding(JCT-VC) under a name High-Efficiency Video Coding (HEVC), aiming, inparticular at improvements of efficiency regarding the high-resolutionvideo coding.

A video signal input to an encoder is a sequence of images calledframes, each frame being a two-dimensional matrix of pixels. All theabove-mentioned standards based on hybrid video coding includesubdividing each individual video frame into smaller blocks consistingof a plurality of pixels. The size of the blocks may vary, for instance,in accordance with the content of the image. The way of coding may betypically varied on a per block basis. The largest possible size forsuch a block, for instance in HEVC, is 64×64 pixels. It is then calledthe largest coding unit (LCU). In H.264/MPEG-4 AVC, a macroblock(usually denoting a block of 16×16 pixels) was the basic image element,for which the encoding is performed, with a possibility to furtherdivide it in smaller subblocks to which some of the coding/decodingsteps were applied.

Typically, the encoding steps of a hybrid video coding include a spatialand/or a temporal prediction. Accordingly, each block to be encoded isfirst predicted using either the blocks in its spatial neighborhood orblocks from its temporal neighborhood, i.e. from previously encodedvideo frames. A block of differences between the block to be encoded andits prediction, also called block of prediction residuals, is thencalculated. Another encoding step is a transformation of a block ofresiduals from the spatial (pixel) domain into a frequency domain. Thetransformation aims at reducing the correlation of the input block.Further encoding step is quantization of the transform coefficients. Inthis step the actual lossy (irreversible) compression takes place.Usually, the compressed transform coefficient values are furthercompacted (losslessly compressed) by means of an entropy coding. Inaddition, side information necessary for reconstruction of the encodedvideo signal is encoded and provided together with the encoded videosignal. This is for example information about the spatial and/ortemporal prediction, amount of quantization, etc.

FIG. 1 is an example of a state of the art hybrid coder 100, as forexample a typical H.264/MPEG-4 AVC and/or HEVC video encoder. Asubtractor 105 first determines differences e between a current block tobe encoded of an input video image (input signal s) and a correspondingprediction block ŝ, which is used as a prediction of the current blockto be encoded. The prediction signal may be obtained by a temporal or bya spatial prediction 180. The type of prediction can be varied on a perframe basis or on a per block basis. Blocks and/or frames predictedusing temporal prediction are called “inter”-encoded and blocks and/orframes predicted using spatial prediction are called “intra”-encoded.Prediction signal using temporal prediction is derived from thepreviously encoded images, which are stored in a memory. The predictionsignal using spatial prediction is derived from the values of boundarypixels in the neighboring blocks, which have been previously encoded,decoded, and stored in the memory.

The difference e between the input signal and the prediction signal,denoted prediction error or residual, is transformed 110 resulting incoefficients, which are quantized 120. Entropy encoder 190 is thenapplied to the quantized coefficients in order to further reduce theamount of data to be stored and/or transmitted in a lossless way. Thisis mainly achieved by applying a code with code words of variable lengthwherein the length of a code word is chosen based on the probability ofits occurrence.

Within the video encoder 100, a decoding unit is incorporated forobtaining a decoded (reconstructed) video signal s′. In compliance withthe encoding steps, the decoding steps include dequantization andinverse transformation 130. The so obtained prediction error signal e′differs from the original prediction error signal due to thequantization error, called also quantization noise. A reconstructedimage signal s′ is then obtained by adding 140 the decoded predictionerror signal e′ to the prediction signal ŝ. In order to maintain thecompatibility between the encoder side and the decoder side, theprediction signal ŝ is obtained based on the encoded and subsequentlydecoded video signal which is known at both sides the encoder and thedecoder.

Due to the quantization, quantization noise is superposed to thereconstructed video signal. Due to the block-wise coding, the superposednoise often has blocking characteristics, which result, in particularfor strong quantization, in visible block boundaries in the decodedimage. Such blocking artifacts have a negative effect upon human visualperception. In order to reduce these artifacts, a deblocking filter 150is applied to every reconstructed image block. The deblocking filter isapplied to the reconstructed signal s′. Deblocking filter generallysmoothes the block edges leading to an improved subjective quality ofthe decoded images. Moreover, since the filtered part of an image isused for the motion compensated prediction of further images, thefiltering also reduces the prediction errors, and thus enablesimprovement of coding efficiency.

After a deblocking filter, an adaptive loop filter 160 may be applied tothe image including the already deblocked signal s″ for improving thepixel wise fidelity (“objective” quality). The adaptive loop filter(ALF) is used to compensate image distortion caused by compression.Typically, the adaptive loop filter is a Wiener Filter, as shown in FIG.1, with filter coefficiency determined such that the mean square error(MSE) between the reconstructed s′, and source images s is minimized.The coefficients of ALF may be calculated and transmitted on a framebasis. ALF can be applied to the entire frame (image of the videosequence) or the local areas (blocks). An additional side informationindicating which areas are to be filtered may be transmitted(block-based, frame-based or quadtree-based).

In order to be decoded, inter-encoded blocks require also storing thepreviously encoded and subsequently decoded portions of image(s) in areference frame buffer (not shown). An inter-encoded block is predicted180 by employing motion compensated prediction. First, a best-matchingblock is found for the current block within the previously encoded anddecoded video frames by a motion estimator. The best-matching block thenbecomes a prediction signal and the relative displacement (motion)between the current block and its best match is then signalized asmotion data in the form of three-dimensional motion vectors within theside information provided together with the encoded video data. Thethree dimensions consist of two spatial dimensions and one temporaldimension. In order to optimize the prediction accuracy, motion vectorsmay be determined with a spatial sub-pixel resolution e.g. half pixel orquarter pixel resolution. A motion vector with spatial sub-pixelresolution may point to a spatial position within an already decodedframe where no real pixel value is available, i.e. a sub-pixel position.Hence, spatial interpolation of such pixel values is needed in order toperform motion compensated prediction. This may be achieved by aninterpolation filter (in FIG. 1 integrated within Prediction block 180).

For both, the intra- and the inter-encoding modes, the differences ebetween the current input signal and the prediction signal aretransformed 110 and quantized 120, resulting in the quantizedcoefficients. Generally, an orthogonal transformation such as atwo-dimensional discrete cosine transformation (DCT) or an integerversion thereof is employed since it reduces the correlation of thenatural video images efficiently. After the transformation, lowerfrequency components are usually more important for image quality thenhigh frequency components so that more bits can be spent for coding thelow frequency components than the high frequency components. In theentropy coder, the two-dimensional matrix of quantized coefficients isconverted into a one-dimensional array. Typically, this conversion isperformed by a so-called zig-zag scanning, which starts with theDC-coefficient in the upper left corner of the two-dimensional array andscans the two-dimensional array in a predetermined sequence ending withan AC coefficient in the lower right corner. As the energy is typicallyconcentrated in the left upper part of the two-dimensional matrix ofcoefficients, corresponding to the lower frequencies, the zig-zagscanning results in an array where usually the last values are zero.This allows for efficient encoding using run-length codes as a partof/before the actual entropy coding.

FIG. 2 illustrates a state of the art decoder 200 according to theH.264/MPEG-4 AVC or HEVC video coding standard. The encoded video signal(input signal to the decoder) first passes to entropy decoder 990, whichdecodes the quantized coefficients, the information elements necessaryfor decoding such as motion data, mode of prediction etc. The quantizedcoefficients are inversely scanned in order to obtain a two-dimensionalmatrix, which is then fed to inverse quantization and inversetransformation 230. After inverse quantization and inversetransformation 230, a decoded (quantized) prediction error signal e′ isobtained, which corresponds to the differences obtained by subtractingthe prediction signal from the signal input to the encoder in the caseno quantization noise is introduced and no error occurred.

The prediction signal is obtained from either a temporal or a spatialprediction 280. The decoded information elements usually further includethe information necessary for the prediction such as prediction type inthe case of intra-prediction and motion data in the case of motioncompensated prediction. The quantized prediction error signal in thespatial domain is then added with an adder 240 to the prediction signalobtained either from the motion compensated prediction or intra-frameprediction 280. The reconstructed image s′ may be passed through adeblocking filter 250, sample adaptive offset processing, and anadaptive loop filter 260 and the resulting decoded signal is stored inthe memory 270 to be applied for temporal or spatial prediction of thefollowing blocks/images

When compressing and decompressing an image, the blocking artifacts aretypically the most annoying artifacts for the user. The deblockingfiltering helps to improve the perceptual experience of the user bysmoothing the edges between the blocks in the reconstructed image. Oneof the difficulties in deblocking filtering is to correctly decidebetween an edge caused by blocking due to the application of a quantizerand between edges which are part of the coded signal. Application of thedeblocking filter is only desirable if the edge on the block boundary isdue to compression artifacts. In other cases, by applying the deblockingfilter, the reconstructed signal may be despaired, distorted. Anotherdifficulty is the selection of an appropriate filter for deblockingfiltering. Typically, the decision is made between several low passfilters with different frequency responses resulting in strong or weaklow pass filtering. In order to decide whether deblocking filtering isto be applied and to select an appropriate filter, image data in theproximity of the boundary of two blocks are considered.

To summarize, state of the art hybrid video coders, see e.g. FIG. 1,apply block-wise prediction and block-wise prediction error coding. Theprediction error coding includes a quantization step. Due to thisblock-wise processing, so called blocking artifacts occur, especially inthe case of coarse quantization. A blocking artifact is associated witha large signal change at a block edge. These blocking artifacts are veryannoying for the viewer. In order to reduce these blocking artifacts,deblocking filtering is applied, e.g. in the H.264/MPEG-4 AVC videocoding standard or in the HM, which is the test model of the HEVC videocoding standardization activity. Deblocking filters decide for eachsample at a block boundary if it is filtered or not and apply a low passfilter in the case it is decided to filter. The aim of this decision isto filter only those samples, for which the large signal change at theblock boundary results from the quantization applied in the block-wiseprocessing. The result of this filtering is a smoothed signal at theblock boundary. The smoothed signal suppresses or reduces the blockingartifacts. Those samples, for which the large signal change at the blockboundary belongs to the original signal to be coded, should not befiltered in order to keep high frequencies and thus the visualsharpness. In the case of wrong decisions, the image is eitherunnecessarily smoothened or remains blocky.

According to the above, it is desirable to reliably judge whether adeblocking filtering needs to be applied at a block boundary betweenadjacent image blocks or not. The H.264/MPEG-4 AVC standard providesdecision operations for the deblocking filtering on a block basis forthe pixels close to the boundary in each individual pixel line, i.e.,pixel row or pixel column respectively, at a block boundary. In general,the block size of the image blocks for which deblocking filteringprocessing is conducted in the H.264/MPEG-4 AVC standard is an 8 by 8pixel block. It is noted, that for other purposes the smallest blocksize may be different, as, for example, prediction is supporting 4 by 4blocks.

FIG. 3 illustrates the decisions for horizontal filtering of a verticalboundary/edge for each individual pixel line according to H.264/MPEG-4AVC. FIG. 3 depicts four 8 by 8 pixel image blocks, the previouslyprocessed blocks 310, 320, 340 and the current block 330. At thevertical boundary between previously processed block 340 and currentblock 330 it is decided, whether deblocking filtering is applied or not.The pixel values of the pixel lines running perpendicular to thevertical boundary serve as a basis for decision for each individualpixel line. In particular, the pixel values in the marked area of eachpixel line, as for instance the marked area 350 of the 5th pixel line,are the basis for the filtering decision.

Similarly, as shown in FIG. 4, decisions for vertical filtering of ahorizontal boundary/edge are performed for each individual column ofpixels. For instance, for the fifth column of the current block 430, thedecision on whether to filter or not, the pixels of this column close tothe boundary to the previously processed block 420 is performed based onthe pixels marked by a dashed rectangle 450.

The decision process for each sample of either each individual pixelcolumn or each individual pixel line, at the boundary is performed byutilizing pixel values of the adjacent blocks as shown in FIG. 5. InFIG. 5, block p represents the previously processed block 340 or 440 asshown in FIG. 3 or FIG. 4 with the pixel values p0, p1 and p2 of oneline (row or column). Block q represents the current block 330 or 430,as in FIG. 3 or FIG. 4, with the pixel values q0, q1 and q2 in the sameline. Pixel q0 is the pixel in the line closest to the boundary with theblock q. Pixel q1 is the pixel in the same line, second closest to theboundary with the block q, etc. In particular, the pixels values p0 andq0 of the pixel line are filtered, if the following conditions aresatisfied:

|p ₀ −q ₀|<α(QP+Offset_(A)),

|p ₁ −p ₀|<β(QP+Offset_(B)), and

|q ₁ −q ₀|<β(QP+Offset_(B)),

wherein, QP is a quantization parameter, Offset_(A) and Offsett_(B) areslice level offsets, and β is chosen to be smaller than α. Further,pixel p1 of the line is filtered, if additionally

|p ₂ −p ₀|<β(QP+Offset_(B)).

Further, the pixel of a pixel line or pixel column corresponding to thepixel value q1 is filtered if additionally

|q ₂ −q ₀|<β(QP+Offset_(B)).

According to H.264/MPEG-4 AVC, for each individual pixel line (row orcolumn for the respective horizontal and vertical deblocking filtering),decision operations as above are performed. The filtering can beswitched on/off for each individual pixel line which is associated witha high accuracy for the deblocking decision. However, this approach isalso associated with a high computational expense.

A decision process for application of a deblocking filtering with alower computational expense as for the above mentioned H.264/MPG-4 AVCstandard, is suggested in “High Efficiency Video Coding (HEVC) textspecification Working Draft 1” (HM deblocking filter, JCTVC-C403),freely available underhttp://wftp3.itu.int/av-arch/jctvc-site/2010_10_C_Guangzhou/, which isincorporated herein by reference. Here, one deblocking filtering on/offdecision is applied for the entire block boundary between two adjacentimage blocks based only on information of pixel lines in the block. Alsohere the block size of the image blocks for which deblocking filteringprocessing is conducted is an 8 by 8 pixel.

The decision for horizontal filtering of a vertical edge/boundaryaccording to JCTVC-C403 is described in the following by referring toFIGS. 6, 8 and 9. FIG. 6 depicts four 8 by 8 pixel blocks, thepreviously processed blocks 610, 620, 640 and the current block 630. Thevertical boundary between the previous block 640 and the current block630 is the boundary for which it is decided, whether deblockingfiltering is to be applied or not. The vertical boundary extends over aboundary segment corresponding to 8 lines (rows) 660. The 3^(rd) and the6^(th) pixel line, which are oriented perpendicular to the verticalboundary, serve as a basis for a deblocking filtering decision. Inparticular, the pixel values in the marked area 650 of the 3^(rd) andthe 6^(th) pixel line are used as a basis for the filtering decision.Hence, the filtering decision of the entire boundary corresponding tothe segment of 8 lines 660, will be based on only a subset of two out of8 pixel lines of the block.

Similarly, referring to FIG. 7, the decision for vertical filtering of ahorizontal edge/boundary according to JCTVC-C403 is based on the pixelvalues of only two pixel columns 760 out of the segment of 8 columns750, which constitutes the horizontal boundary.

FIG. 8 shows a matrix of pixel values, which corresponds to parts of theprevious block 640 and the current block 630 of FIG. 6. The pixel valuesin the matrix are p_(i,j) and q_(i,j), with i being an index varyingperpendicular to the boundary between the blocks and with j being anindex varying along to the boundary between the blocks. Index i in FIG.8 varies only in the range from 0 to 3, corresponding to the pixelpositions within a line to be filtered, which are employed for thedecision and/or filtering. The remaining pixel positions of the previousand the current block are not shown. Index j in FIG. 8 varies in therange from 0 to 7, corresponding to the 8 pixel rows in the block, thevertical boundary of which is to be filtered. The two pixel lines 820with indexes j=2 and j=5, which correspond to the respective 3^(rd) andthe 6^(th) pixel lines, are used as a basis for the filtering decision(on/off decision) for the entire block boundary and are marked withdashed lines. In order to decide whether the segment of 8 pixel lines,which correspond to the entire boundary, is filtered, the followingcondition is evaluated:

|p2₂−2·p1₂ +p0₂ |+|q2₂−2·q1₂ +q0₂ |+|p2₅−2·p1₅ +p0₅ |+|q2₅−2·q1₅+q0₅|<β,

wherein β is a threshold value. If the above condition is true, it isdecided that the filtering is to be applied to all 8 lines of theboundary.

This decision process is further depicted in FIG. 9. When the upperequation is separated into a term d_(1,v), containing only pixel valuesof the pixel line with index j=2 and a term d_(2,v), containing onlypixel values of the line with index j=5, the decision for filtering canbe rewritten as:

d _(1,v) +d _(2,v)<β,

wherein

d _(1,v) =|p2₂−2·p1₂ +p0₂ |+|q2₂−2·q1₂ +q0₂|

and

d _(2,v) =|p2₅−2·p1₅ +p0₅ |+|q2₅−2·q1₅ +q0₅|

Hence, by use of the two values d_(1,v) and d_(2,v), it is decided bythe threshold operation whether the entire vertical boundary is to befiltered or not. The index v is used herein to indicate that a decisionfor a vertical boundary is assessed.

FIG. 8 shows a matrix of pixel values forming boundary portions of twoneighbouring blocks A and B. It is noted that this boundary may also bea horizontal boundary, so that the block A is a previously processedblock and block B is the current block, block A being the top neighbourof block B. This arrangement corresponds to parts of the previous block720 and the current block 730 in FIG. 7. The pixel values in the matrixare p_(i,j) and q_(i,j), with i being an index varying perpendicular tothe boundary between the blocks, the index i ranging from 0 to 3 in thisexample corresponding to only the part of the block A and B shown, andwith the index j varying along the boundary between the blocks A and B,ranging from 0 to 7 corresponding to the number of lines (in third casecolumns) to be processed by deblocking filtering. In this context,“processing” or “deblocking processing” includes deciding whetherdeblocking filtering is to be applied or not and/or selection of thefilter type. The type of filter here refers to a weak, strong or nofilter for filtering pixels around the boundary in a particular line ofthe block. The derivation process of boundary filtering strength isdescribed, for instance, in section 8.1.6 of the above mentioned “HighEfficiency Video Coding (HEVC) text specification Working Draft 1”. Inparticular, when it is decided that the block is to be filtered, anindividual decision is performed for each line for deciding whether astrong filter or a weak filter is to be applied. If it is decided that aweak filter is to be applied, it is tested whether it is to be appliedto the line at all. A strong filter in this sense is applied to morepixels around the boundary in the pixel line than the weak filter. Ingeneral, a strong filter is a filter with a narrower pass-band than theweak filter.

The two pixel columns 820 with indexes j=2 and j=5, which correspond tothe 3^(rd) and the 6^(th) pixel column, are used as a basis for thefiltering decision and are marked with dashed lines. The horizontalboundary is filtered if

|p2₂−2·p1₂ +p0₂ |+|q2₂−2·q1₂ +q0₂ |+|p2₅−2·p1₅ +p0₅ |+|q2₅−2·q1₅+q0₅|<β,

wherein β is again a threshold value. If the above decision is true,filtering is applied to all 8 columns of the horizontal boundary, whichcorresponds to entire boundary. This decision process is furtherdepicted in FIG. 10. When the upper equation is separated into a termd_(1,h) containing only pixel values of the pixel column with index j=2and a term d_(2,h), containing only pixel values of the pixel columnwith index j=5, the decision for filtering can be rewritten as:

d _(1,h) +d _(2,h)<β,

wherein

d _(1,h) =|p2₂−2·p1₂ +p0₂ |+|q2₂−2·q1₂ +q0₂|

and

d _(2,h) =|p2₅−2·p1₅ +p0₅ |+|q2₅−2·q1₅ +q0₅|.

Hence, by the use of the two values d_(1,h) and d_(2,h), it is decidedby the threshold operation if the entire horizontal boundary is filteredor not. The index h is hereby used to indicate that a decision for ahorizontal boundary is assessed.

To summarize, according to JVCT-D403, the filtering can be switchedon/off for the entire boundary based on only two pixel lines or pixelcolumns perpendicular to that boundary. For only two positions of eachsegment of 8 lines/columns, a decision process is performed. Thefiltering can be switched on/off for each segment of 8 lines/columns,corresponding to the entire block. This is associated with a lowercomputational expense but also with a lower accuracy of the decisions.

In contribution JCTVC-D263, “Parallel deblocking Filter”, Daegu, January2011, freely available athttp://wftp3.itu.int/av-arch/jctvc-site/2011_01_D_Daegu/ which isincorporated herein by reference, the decision operations for deblockingfiltering of a block are performed similarly to JCTVC-C403: Onedeblocking filtering on/off decision is applied for the entire blockboundary based only on pixel values of two pixel rows, or pixel columnsrespectively, of the two vertically or horizontally adjacent imageblocks. However, the difference between the two approaches is that thepixel rows, or pixel columns respectively, which are used as a basis forthe decision whether the boundary is filtered or not, have a differentposition in the block.

The decision for horizontal filtering of a vertical boundary/edgeaccording to JCTVC-D263 is briefly described in the following byreferring to FIGS. 11 and 13. In FIG. 11, the pixel lines used as abasis for deciding on whether to filter or not, are the 4^(th) and5^(th) lines 1160 at the boundary between the previous 1140 and thecurrent block 1130. The entire vertical boundary corresponds to asegment of 8 lines 1150.

FIG. 13 shows a matrix of pixel values forming parts of the blocks A andB around a common boundary. The blocks A and B correspond to theprevious block 1140 and the current block 1130 of FIG. 11, respectively.The pixel values in the matrix are and with i being an index varyingperpendicular to the boundary between the blocks and ranging from 0 to3, and with j being an index varying along to the boundary between theblocks and ranging from 0 to 7. The two pixel lines 1320 with indexesj=3 and j=4, which correspond to the 4^(th) and the 5^(th) pixel line,are used as a basis for the filtering decision(s) and are marked withdashed lines. The following condition is evaluated in order to judgewhether to filter or not the pixels close to the boundary in the currentblock:

|p2₃−2·p1₃ +p0₃ |+|q2₃−2·q1₃ +q0₃ |+|p2₄−2·p1₄ +p0₄ |+|q2₄−2·q1₄+q0₄|<β,

wherein β is a threshold value. If the above decision is true, filteringand/or further decision is performed for all lines of the boundary whichcorresponds to a segment comprising 8 lines. When the upper equation isseparated into a term d_(1,v), containing only pixel values of the pixelline with index j=3 and a term d_(2,v), containing only pixel values ofthe line with index j=4, the decision for filtering can be rewritten as:

d _(1,v) +d _(2,v)<β,

wherein

d _(1,v) =|p2₃−2·p1₃ +p0₃ |+|q2₃−2·q1₃ +q0₃|

and

d _(2,v) =|p2₄−2·p1₄ +p0₄ |+|q2₄−2·q1₄ +q0₄|.

Hence, by the use of the two values d_(1,v) and d_(2,v), it is decidedby the threshold operation if all 8 lines of the corresponding segmentare filtered or not. The index v is hereby used to indicate that adecision for a vertical boundary is assessed.

Similarly, as shown in FIG. 12, the decision for vertical filtering of ahorizontal edge/boundary between a current block 1230 and a previousblock 1220 according to JCTVC-D263 is based on the pixel values of onlytwo columns 1260 out of the segment 1250 of pixels from 8 columns whichconstitutes the horizontal boundary between the blocks 1230 and 1220.

FIG. 13 may be also seen as corresponding to parts of the previous block1220 and the current block 1230 of FIG. 12. The pixel values in thematrix are p_(i,j) and q_(i,j), with i being an index varyingperpendicular to the boundary between the blocks, ranging from 0 to 3and with j being an index varying along to the boundary between theblocks, ranging from 0 to 7. The two pixel columns 1320 with indexes j=3and j=4, which in this example correspond to the 4^(th) and the 5^(th)pixel column, are used as a basis for the filtering decision and aremarked with dashed lines. Accordingly, the horizontal boundary isfiltered when

|p2₃−2·p1₃ +p0₃ |+|q2₃−2·q1₃ +q0₃ |+|p2₄−2·p1₄ +p0₄ |+|q2₄−2·q1₄+q0₄|<β,

wherein β is a threshold value. If the above condition is true,filtering is applied to all columns of the boundary corresponding to onesegment which is composed of 8 columns. When the upper equation isseparated into a term d_(1,h), containing only pixel values of thecolumn with index j=3 and a term d_(2,h), containing only pixel valuesof the column with index j=4, the decision for filtering can berewritten as:

d _(1,h) +d _(2,h)<β,

wherein

d _(1,h) =|p2₃−2·p1₃ +p0₃ |+|q2₃−2·q1₃ +q0₃|

and

d _(2,h) =|p2₄−2·p1₄ +p0₄ |+|q2₄−2·q1₄ +q0₄|.

Hence, by using the two values d_(1,h) and d_(2,h), it is decided by thethreshold operation whether all 8 columns of the segment 1010 arefiltered or not. The index h is hereby used to indicate that a decisionfor a horizontal boundary is assessed.

To summarize, similarly to the JVCT-C403, according to JVCT-D263, thefiltering can be switched on/off for the entire boundary segment basedon only two pixel lines or pixel columns from this segment. For only twopositions of each segment of 8 lines (rows or columns), a decisionprocess is performed. Thus, the filtering can be switched on/off foreach segment of 8 lines/columns. This is associated with a lowcomputational expense but also with a low accuracy of the decisions. Anadvantage of JCTVC-D263 over JCTVC-C403 is that the use of other samplesallows a higher degree of a parallel processing. However, bothapproaches JCTVC-C403 and JCTVC-D263 provide a lower accuracy ofdecisions in comparison with, for example, H.264/MPEG-4 AVC.

In H.264/MPEG-4 AVC, the decisions are performed as shown in FIG. 2 toFIG. 5. At each pixel position at a block boundary, individual valuesare calculated using samples adjacent to the block boundary. Based onthese individual values, individual decision operations are performed ateach position of (for each line perpendicular to) the block boundary.This is associated with a high computational expense while providing ahigh accuracy of the decisions. In JCTVC-C403, pixels at the block edgesform segments of 8 lines/columns (corresponding to the smallest blocksize used for the deblocking filtering) as shown in FIG. 6 and FIG. 7.For each segment of 8 lines/columns, values are calculated only for asubset of positions, in the examples above for only two positions ratherthan for all 8 positions. Based on these values, one single decision isperformed whether to filter all 8 lines/columns of the segment or not.Compared to H.264/MPEG-4 AVC the computational expense is reduced sinceless values are calculated. The term value refers to the measure basedon values of the pixels in a line close to the boundary such as d_(1,v)and d_(2,v) or d_(1,h) or d_(2,h) as shown above. In addition, thememory bandwidth is reduced since for the calculation of values, lesssamples need to be accessed from the memory. However, also the accuracyof the decisions is reduced compared to the accuracy of the decisions inH.264/MPEG-4 AVC. In JCTVC-D263, the calculation of values and thedecision operations are performed similar to the JCTVC-C403. Thedifference is that samples at other positions of the segments of 8lines/columns are used to calculate the values. The use of these othersamples allows a higher degree a parallel processing. Compared toJCTVC-C403, the computational expense as well as the memory bandwidth isthe same. However, the accuracy of the decisions is further reduced.Details are explained in FIG. 11 to FIG. 13. Thus, the known approachesare either associated with a high computational expense and high memorybandwidth or with a low accuracy of the decisions. A low accuracy of thedecisions, on the other hand, may result to a low coding efficiency.High computational expense and high memory bandwidth may both lead tohigh implementation costs.

SUMMARY OF THE INVENTION

In view of the above problems with the existing deblocking filteringapproaches, the present invention aims to provide a more efficientdeblocking filtering with improved accuracy and reduced computationalexpenses.

It is the particular approach of the present invention to judge whetheror not to apply a deblocking filter to segments of the boundary of ablock by judging individually for each segment of the boundary based onpixels comprised in a subset of pixel lines of the block.

According to an aspect of the present invention, a method for deblockingprocessing of an image divided into blocks, of which the boundaries areto be processed, is provided, wherein each block is composed of pixellines perpendicular to a boundary with an adjacent block, the methodcomprising the steps of judging whether or not to apply a deblockingfilter to segments of the boundary of the block by judging individuallyfor each segment of the boundary based on pixels comprised in a subsetof pixel lines of the block, and applying or not applying the deblockingfilter to the segments of the boundary according to the result of therespective individual judgements

According to another aspect of the present invention, an apparatus fordeblocking processing of an image divided into blocks, of which theboundaries are to be processed, is provided, wherein each block iscomposed of pixel lines perpendicular to a boundary with an adjacentblock, the apparatus comprising a judging unit configured to judgewhether or not to apply a deblocking filter to segments of the boundaryof the block by judging individually for each segment of the boundarybased on pixels comprised in a subset of pixel lines of the block, and adeblocking filtering unit configured to apply or not apply thedeblocking filter to the segments of the boundary according to theresult of the respective individual judgements.

The above and other objects and features of the present invention willbecome more apparent from the following description and preferredembodiments given in conjunction with the accompanying drawings inwhich:

FIG. 1 is a block diagram illustrating an example of a state of the arthybrid coder;

FIG. 2 is a block diagram illustrating an example of a state of the arthybrid decoder;

FIG. 3 is a schematic drawing illustrating the decisions for horizontaldeblocking filtering of a vertical edge according to H.264/MPEG-4 AVC;

FIG. 4 is a schematic drawing illustrating decisions for verticaldeblocking filtering of a horizontal edge according to H.264/MPEG-4 AVC;

FIG. 5 is a schematic drawing illustrating the decision process for eachsample at the block boundary whether to filter or not according toH.264/MPEG-4AVC;

FIG. 6 is a schematic drawing illustrating the decision process for eachsample at the block boundary whether to filter or not according toJCTVC-C403 for horizontal filtering of a vertical edge;

FIG. 7 is a schematic drawing illustrating a decision process for eachsample at the block boundary whether to filter or not according toJCTVC-C403 for vertical filtering of a horizontal edge;

FIG. 8 is a schematic drawing illustrating the decision process for eachsegment of 8 lines/columns whether to filter or not according toJCTVC-C403;

FIG. 9 is a schematic drawing illustrating the decision process for eachsample at the block boundary whether to filter or not according toJCTVC-C403 for horizontal filtering of a vertical edge;

FIG. 10 is a schematic drawing illustrating a decision process for eachsample at the block boundary whether to filter or not according toJCTVC-C403 for vertical filtering of a horizontal edge as according toFIG. 7;

FIG. 11 is a schematic drawing illustrating the decision process foreach sample at the block boundary whether to filter or not according toJCTVC-D263 for horizontal filtering of a vertical boundary;

FIG. 12 is a schematic drawing illustrating the decision process foreach sample at the block boundary whether to filter or not according toJCTVC-D263 for vertical filtering of a horizontal boundary;

FIG. 13 is a schematic drawing illustrating the decision process foreach segment of 8 lines/columns whether to filter or not according toJCTVC-D263;

FIG. 14 is a schematic drawing illustrating the decision process forhorizontal filtering of a vertical boundary according to an embodimentof the present invention;

FIG. 15 is a schematic drawing illustrating the decisions for verticalfiltering of a horizontal boundary according to an embodiment of thepresent invention;

FIG. 16 is a schematic drawing illustrating the decision process forhorizontal filtering of a vertical boundary according to an embodimentof the present invention;

FIG. 17 is a schematic drawing illustrating the decisions for verticalfiltering of a horizontal boundary according to an embodiment of thepresent invention;

FIG. 18 is a schematic drawing illustrating the decisions for horizontalfiltering of a vertical boundary according to an embodiment of thepresent invention;

FIG. 19 is a schematic drawing illustrating the decision for verticalfiltering of a horizontal boundary according to an embodiment of thepresent invention;

FIG. 20 is a schematic drawing illustrating the decision processaccording to an embodiment of the present invention;

FIG. 21 is a schematic drawing illustrating the decision processaccording to an embodiment of the present invention;

FIG. 22 is a schematic drawing illustrating the decision processaccording to an embodiment of the present invention;

FIG. 23 is a generalized block diagram of the hybrid video encoderaccording to the HM 2.0;

FIG. 24 is an illustration of the signal before and after the deblockingfilter for a region of the example test sequence Kimono;

FIG. 25 is a schematic drawing illustrating vertical edges and thehorizontal edges of an example coding unit (CU) of the size 16×16samples

FIG. 26 shows the notation of a part of a vertical edge for deblocking;

FIG. 27 shows an illustration of the samples used to decide whether tofilter or not according to the HM2.0;

FIG. 28 shows an illustration of the samples used to decide whether tofilter or not similar as in H.264/MPEG-4 AVC;

FIG. 29 shows an illustration of the samples used to decide whether tofilter or not according to an embodiment of the invention;

FIG. 30 shows BD-bit rates and run time ratios of the decisions similaras in H.264/MPEG-4 AVC compared to the reference HM2.0;

FIG. 31 shows BD-bit rates and run time ratios of the decisionscompromising HM2.0 and H.264/MPEG-4 AVC compared to the reference HM2.0;

FIG. 32 illustrates subjective quality of the approach of an embodimentof the present invention compared to the reference with the resultsshown in the table;

FIG. 33 shows the cropped part of a deblocked frame of the test sequenceVidyo3 in the case of the reference HM 2.0. Test case: Low delay, HighEfficiency, QP37;

FIG. 34 shows the cropped part of a deblocked frame of the test sequenceVidyo3 in the case of the proposal. Test case: Low delay, HighEfficiency, QP37;

FIG. 35 shows the cropped part of a deblocked frame of the test sequenceVidyo3 in the case of the reference HM 2.0. Test case: Low delay, HighEfficiency, QP37;

FIG. 36 shows the cropped part of a deblocked frame of the test sequenceVidyo3 in the case of the proposal. Test case: Low delay, HighEfficiency, QP37;

FIG. 37 illustrates the BD-bit rate reduction averaged over all testcases and test sequences versus additional number of required operationsper edge segment compared to the reference HM2.0;

FIG. 38 is a schematic drawing illustrating an overall configuration ofa content providing system for implementing content distributionservices;

FIG. 39 is a schematic drawing illustrating an overall configuration ofa digital broadcasting system;

FIG. 40 is a block diagram illustrating an example of a configuration ofa television;

FIG. 41 is a block diagram illustrating an example of a configuration ofan information reproducing/recording unit that reads and writesinformation from or on a recording medium that is an optical disk;

FIG. 42 is a schematic drawing showing an example of a configuration ofa recording medium that is an optical disk;

FIG. 43A is a schematic drawing illustrating an example of a cellularphone;

FIG. 43B is a block diagram showing an example of a configuration of thecellular phone;

FIG. 44 is a schematic drawing showing a structure of multiplexed data;

FIG. 45 is a drawing schematically illustrating how each of the streamsis multiplexed in multiplexed data;

FIG. 46 is a schematic drawing illustrating how a video stream is storedin a stream of PES packets in more detail;

FIG. 47 is a schematic drawing showing a structure of TS packets andsource packets in the multiplexed data;

FIG. 48 is a schematic drawing showing a data structure of a PMT;

FIG. 49 is a schematic drawing showing an internal structure ofmultiplexed data information;

FIG. 50 is a schematic drawing showing an internal structure of streamattribute information;

FIG. 51 is a schematic drawing showing steps for identifying video data;

FIG. 52 is a schematic block diagram illustrating an example of aconfiguration of an integrated circuit for implementing the video codingmethod and the video decoding method according to each of embodiments;

FIG. 53 is a schematic drawing showing a configuration for switchingbetween driving frequencies;

FIG. 54 is a schematic drawing showing steps for identifying video dataand switching between driving frequencies;

FIG. 55 is a schematic drawing showing an example of a look-up table inwhich the standards of video data are associated with the drivingfrequencies;

FIG. 56A is a schematic drawing showing an example of a configurationfor sharing a module of a signal processing unit;

FIG. 56B is a schematic drawing showing another example of aconfiguration for sharing a module of a signal processing unit;

DETAILED DESCRIPTION

The problem underlying the present invention is based on the observationthat the currently employed approaches for deblocking filtering lead toeither reduced filtering quality or to rather high computationalexpenses.

In order to provide a more efficient filtering approach, according tothe present invention, the decisions related to the deblocking filteringare performed for segments of the blocks to be filtered by thedeblocking filter rather than for the entire blocks. Moreover, thedecisions are performed based on only a subset of the pixels in theblock which are situated at the boundary.

In general, as also described in the background section, the decisionsmay be the decision on whether or not to filter a segment of theboundary and/or whether to apply the filter to pixels at a particulardistance from the boundary (corresponding to the decision about thestrength of the filter), etc.

Herein, a block is a smallest block of pixels (samples) being confinedby boundaries which are processed by deblocking filtering. Theprocessing at each boundary of a block includes decision on whether toapply the filtering and/or what kind of filter to apply and/or applyingor not the filter according to the decision(s). As also described thebackround section, the block size of which the boundaries are processedby deblocking filtering is typically an 8 by 8 pixel similar to H.264and the HEVC standards as JCTVC-D403 and JCTVC-D263. A block may befurther seen as being comprised of pixel lines perpendicular withrespect to a specified boundary of the block.

The term boundary is referring to a logical line separating pixels oftwo neighbouring blocks. The boundary of a smallest block to beprocessed by deblocking filtering, extends over all pixel lines of theblock oriented perpendicular to the boundary and also extends betweentwo other boundaries of the block which are oriented perpendicularly.

A segment is a portion of a block including one or more pixel linesoriented perpendicular to the boundary with pixels to be processed bythe deblocking filter. The segment of the block is a subset of the pixellines of entire block, i.e. a proper partial subset, meaning that itincludes less than all pixel lines of the block. Thus a segment extendsover a certain number of pixel lines in a direction parallel to theboundary. However, a segment does not extend over all pixel lines of ablock. Further, a segment of the boundary corresponds to the portion ofthe boundary where the segment of the block portion is situated at theboundary.

Pixels at the boundary of a block are pixels in a block being situatedclose to the boundary to an adjacent block. Pixels at the boundary mayinclude the pixels directly at (closest to) the boundary, the pixelswhich are second closest to the boundary, and/or the third closest, etc.

The deblocking filtering is typically performed by a 1-dimensionalfilter, vertical or horizontal. The filter is applied orthogonally tothe boundary, in particular, to the pixels at the boundary included in apixel line of the block perpendicular to the boundary.

FIGS. 14 and 16 illustrate the decision process for horizontal filteringof a vertical boundary between two adjacent image blocks according to anembodiment of the present invention. Similarly, FIGS. 15 and 17illustrate the decision process for vertical filtering of a horizontalboundary between two adjacent image blocks according to an embodiment ofthe present invention.

FIG. 14 shows four 8×8 pixel image blocks, namely the previouslyprocessed blocks 1410, 1420, 1440 and the current block 1430 on the lefthand side. Block 1410 is the top left neighbour of the current block1430, block 1420 is the top neighbour of the current block 1430 andblock 1440 is the left neighbour of the current block 1430. The verticalboundary 1450 between the left adjacent block 1440 and the current block1430 is the boundary for which the decision for horizontal filtering iscarried out. This boundary 1450 basically extends between and is at thesame time confined by an upper horizontal boundary 1470 and the lowerhorizontal boundary 1480. The upper 1470 and lower 1480 horizontalboundaries may be filtered vertically. The previous block 1440 and thecurrent block 1430 adjacent to the boundary 1450 are composed of 8 pixellines oriented perpendicular to the boundary. Hence, the verticalboundary for horizontal filtering in FIG. 14 is extending over a segmentof 8 pixel lines 1450. The boundary can be divided into segments,wherein the smallest segment is extending over one pixel line.

In order to decide whether or not to apply deblocking filter to segmentsof the block boundary 1450, pixels from a (proper partial) subset of thepixel lines from the current block 1430 and/or the previous block 1440are used as a basis for decision. As also in the approaches described inthe background section, the pixels from the subset of lines (rows) inthe previous block 1440 and the current block 1430 are the pixels at the(close to) common boundary between these blocks. In the example of FIG.14, two out of eight pixel lines are used for deciding whether or not toapply a deblocking filter to each segment of the boundary. In this casethe 3^(rd) and 6^(th) pixel line is chosen. These two pixel linesrepresent a (proper partial) subset of the 8 pixel lines that theprevious 1440 and the current block 1430 are composed of. Herein, aproper partial subset of pixel lines of a block is defined as any numberof pixel lines which is smaller than the total number of pixel linesthat an image block is composed of. Subsequently, the samples from thesubset of lines, in this case, from the two pixel lines, are used forperforming individual decisions for segments of the boundary, asdepicted on the right hand side of FIG. 14. This is achieved, forinstance, by calculating line decision terms d_(1,v) and d_(2,v) as afunction of the pixels from the subset of lines. The values d_(1,v) andd_(2,v) may be calculated similar as the values d_(1,v) and d_(2,v)according to JCTVC-C403 or JCTVC-D263, as described above. These valuesmay be calculated, for instance, as gradients of the 1^(st) or the2^(nd) order between the neighbouring pixels in each respective of thetwo neighbouring blocks 1440 and 1430, or between pixels from bothblocks 1440 and 1430. These gradients may be calculated as differencesbetween these pixels. Such measures are advantageous for estimating theblocking effect between two blocks.

Further, an individual decision value F_(N), which corresponds to anindividual function of the line decision terms d_(1,v) and d_(2,v), iscompared with a threshold value β for each segment of a number ofsegment from 1 to N:

F _(N)(d _(1,v) ,d _(2,v))<β.

In the case the above condition is true, filtering is applied to theindividual segment of the vertical boundary 1450. It is noted that theline terms d_(1,v) and d_(2,v) do not necessarily have to be calculatedin a separate step. The individual decision value may also be calculatedwithout having precalculated and stored the line decision termsseparately before. In this example, each boundary position correspondingto each line of the block(s) to be filtered is a segment and for each ofthese lines it is decided based on individual function of the pixelsfrom the subset of lines whether the this boundary position is to befiltered or not. This corresponds in this example to interpolation orextrapolation (depending on the segment position) of the individualdecision term based on 1) the pixels of the subset of block lines and 2)on the position of the segment.

FIG. 15 illustrates the decisions for vertical filtering of a horizontalboundary similar to the horizontal filtering of the vertical boundarydescribed above with reference to FIG. 14. Here, instead of the 3^(rd)and the 6^(th) pixel line, the 3^(rd) and the 6^(th) pixel column arethe basis for the filtering decisions. The information obtained from thesubset of lines formed by the 3^(rd) and 6^(th) pixel columnscorresponds to the calculated values, line decision terms, d_(1,h) andd_(2,h). Further, an individual decision value (F_(N)), which is anindividual function of the line decision terms d_(1,h) and d_(2,h), iscompared with a threshold value β for each segment of a number ofsegment from 1 to N:

F _(N)(d _(1,h) ,d _(2,h))<β

In the case the above condition is true, filtering is applied to theindividual segment of the horizontal boundary 1550. In this example,each line may be an individual segment, for which an individual FunctionF_(N) is applied. The function is not necessarily computed as a functionof the line decision terms, it may be also directly computed from theindividual pixels in the subset lines.

FIG. 16 exemplifies a particular solution and implementation for theabove individual functions of the calculated values based on the 3^(rd)and 6^(th) pixel line for individual segments of the boundary. In thiscase, three individual decisions for three respective block (boundary)segments are performed based on respective three individual decisionvalues. In particular, FIG. 16 shows on the right hand side, that forthe first to the third pixel line, value d_(1,v) obtained based on thepixels of the 3^(rd) pixel line is utilized for the following decision:

2·d _(1,v)<β.

In the case the above condition is true, filtering is applied to thesegment extending over the first to the third pixel line of the boundary1650. However, this can also be seen as a same decision for theindividual segments extending over the first, the second or the thirdpixel line respectively. Thus, the individual decision values for thefirst and the second pixel line can be also seen as a nearest neighborinterpolation of the individual decision value of the third segment.This means that the individual decision value used for the line forwhich the line decision term is calculated, is also used for the otherlines within the same segment. For a further segment of the boundary,which corresponds to the fourth and fifth pixel line of the boundary,information from both the third and the sixth pixel line is used. Thevalues d_(1,v) and d_(2,v) are utilized for the following decision:

d _(1,v) +d _(2,v)<β

In the case the above condition is true, filtering is applied to thesegment extending over the fourth and the fifths pixel line of theboundary 1650. However, this can also be seen as a same decision for theindividual segments extending over the fourth or the fifth pixel linerespectively. For another segment of the boundary, which corresponds tothe sixth to the eight pixel position of the boundary, the informationof the sixth pixel line is utilized for the filtering decision. Thevalue d_(2,v) is utilized for the following decision:

2·d _(2,v)<β

In the case the above condition is true, filtering is applied to thesegment being extending over the sixth to the eights pixel line of theboundary 1650. However, this can also be seen as a same decision for theindividual segments extending over the sixth, the seventh or the eighthpixel line, respectively. Nevertheless, in order to achieve theadvantages of the present invention, decision are performed for at leasttwo segments of the boundary individually and at least two individuallycalculated decision values are applied in the decision process.

FIG. 17 shows, corresponding to FIG. 16, a particular solution andimplementation for the above individual functions of the calculatedvalues based on the 3^(rd) and 6^(th) pixel column for each segment ofthe horizontal boundary. In particular, FIG. 17 shows on the right handside, that for the first to the third pixel column, the value d_(1,h)obtained based on the 3^(rd) pixel column is utilized for the followingdecision:

2·d _(1,h)<β.

In the case the above condition is true, filtering is applied to thesegment being extending over the first to the third pixel column of theboundary 1750. However, this can also be seen as a same decision for theindividual segments extending over the first, the second or the thirdpixel column respectively. For a further segment of the boundary, whichcorresponds to the fourth and fifth pixel column of the boundary,information from the third and the sixth pixel column is used.Correspondingly, the values d_(1,h) and d_(2,h) are utilized for thefollowing decision:

d _(1,h) +d _(2,h)<β

In the case the above condition is true, filtering is applied to thesegment being extending over the fourth and the fifths pixel column ofthe boundary 1750. However, this can also be seen as a same decision forthe individual segments extending over the fourth or the fifth pixelcolumn respectively. For another segment of the boundary, whichcorresponds to the sixth to the eight pixel position of the boundary,the information of the sixth pixel column is utilized for the filteringdecision. Correspondingly, the value d_(2,h) is utilized for thefollowing decision:

2·d _(2,h)<β

In the case the above condition is true, filtering is applied to thesegment being extending over the sixth to the eights pixel column of theboundary 1750. However, this can also be seen as a same decision for theindividual segments extending over the sixth, the seventh or the eighthpixel column, respectively.

To summarize according to the present embodiment of the invention, theprocess of judging whether or not to apply a deblocking filter tosegments of the boundary of the block is conducted by judgingindividually for each segment of the boundary based on pixels comprisedin a subset of pixel lines of the block and applying or not applying thedeblocking filter to the segments of the boundary according to theresult of the respective individual judgements.

Further, when judging whether or not to apply a deblocking filter tosegments of the boundary of the block, an individual decision value foreach segment of the boundary by using pixel values of pixels comprisedin at least one pixel line of the subset of the pixel lines of the blockis obtained and compared with a threshold value for each individualsegment of the boundary.

At least one of the individual decision values from the obtainedindividual decision values is based on only one pixel line of the subsetof the pixel lines. At further individual decision value based on onepixel line of the subset of the pixel lines may be obtained by applyingnearest neighbour interpolation to the firstly obtained decision value.

Further, at least another individual decision value may be based on atleast two pixel lines of the subset of the pixel lines. This latterindividual decision value may be a linear combination of individualdecision values which are based on one pixel line of the subset of thepixel lines.

According to another embodiment of the present invention, the process ofjudging whether or not to apply a deblocking filter to segments of theboundary of the block is based on pixels comprised in pixel lines ofanother block, which is adjacent to the block and situated acrossanother boundary which perpendicular to the boundary which is currentlyprocessed, in addition to being based on pixels comprised in a subset ofthe pixel lines of the block. Hence, in order to perform individualdecisions for segments of the boundary, not only information comprisedin the subset of pixel lines of the block is used, but also pixel lines(rows or columns) of adjacent or previous blocks respectively. This isillustrated for example in FIG. 18 for a decision for horizontalfiltering of a vertical boundary/edge. In particular, as an example, inFIG. 18 additional information for individual decisions of segments isobtained from pixels of the 3^(rd) line of the two upper previous blocks1810, 1820 of the four image blocks 1810, 1820, 1830 and 1840, as shownon the left hand side. These pixel values of the the 3^(rd) line of theprevious blocks 1810 and 1820 is used for calculating the decision termd_(0,v) in addition to calculating the decision terms d_(1,v), d_(2,v)by pixel values of a subset of pixel lines of the current 1830 andprevious block 1840. As shown on the right hand side, individualdecisions for segments are now based on the calculated values d_(1,v)and d_(2,v) being itself based on the 3^(rd) and 6^(th) pixel lines ofthe previous 1840 and the current block 1830 adjacent to the verticalboundary, and also on the calculated values d_(0,v) being based on the3^(rd) pixel line of the previous blocks 1810 and 1820. Subsequently, anindividual decision value F_(N), which is an individual function of thepreviously obtained information, the calculated values (decision terms)d_(1,v), d_(2,v), and d_(0,v), is compared with a threshold value β foreach segment of a number of segment from 1 to N:

F _(N)(d _(0,v) ,d _(1,v) ,d _(2,v))<β.

In the case the above condition is true, filtering is applied to theindividual segment of the vertical boundary. Similarly, this principlecan be also applied for decisions for vertical filtering of a horizontalboundary as illustrated in FIG. 19.

According to another embodiment of the present invention, in the processof judging whether or not to apply a deblocking filter to segments ofthe boundary of the block, the pixel lines serving as a basis forjudging are regularly distributed in a direction parallel to theboundary which is processed. As example of this embodiment, FIG. 20illustrates the decisions for horizontal filtering of a verticalboundary between the previous block 2040 and the current block 2030.Here, the pixel lines which are used as a basis for decisions ondeblocking filtering of segments of the vertical boundary are spacedregularly in a direction parallel to the vertical boundary. In otherwords, the pixel lines for calculating, for example a line decision termd, have a same regular distance from each other. In the example of FIG.20, all pixel line used as a basis for decisions for deblockingfiltering are spaced apart by three pixel lines, which are not used as abasis for decisions for deblocking filtering. This embodiment may bebeneficial for achieving a more efficient deblocking filtering decision.

In another embodiment of the present invention, in the process ofjudging whether or not to apply a deblocking filter to segments of theboundary of the block, individual decision values based on one pixelline of the subset of the pixel lines are interpolated linearly in orderto obtain individual decision values for each segment of the boundarywhich is then compared to a threshold value. FIGS. 21 and 22 illustratethe decision for vertical filtering of a horizontal edge according tothis embodiment. In particular, the subset of pixel lines which is usedas a basis for individual decisions for each segment of the boundary isnow constituted by four pixel lines out of eight pixel lines that theblock is composed of. In FIG. 21, this is the 1^(rst), 3^(rd), 6^(th)and 8^(th) pixel line. Based thereon, the values d_(1,v), d_(2,v),d_(3,v) and d_(4,v) (line decision term) are calculated and used forobtaining the individual decision values, as shown in FIG. 22, for eachsegment constituting the vertical boundary between the previous block2140 and the current block 2130. In particular, the condition forjudging whether or not to apply a deblocking filter at the first segmentwhich corresponds to the 1^(rst) pixel line is the following:

2·d _(1,v)<β.

The condition for judging whether or not to apply a deblocking filterfor the second segment which corresponds to the second pixel line is thefollowing:

d _(1,v) +d _(2,v)<β

The condition for judging whether or not to apply a deblocking filterfor the third segment which corresponds to the third pixel line is thefollowing:

2·d _(2,v)<β

The condition for judging whether or not to apply a deblocking filterfor the fourth segment which corresponds to the fourth pixel line is thefollowing:

(4·d _(2,v)+2·d _(3,v))/3<β

Alternatively, the condition for judging whether or not to apply adeblocking filter for the fourth segment which corresponds to the fourthpixel line could be the following:

(4·d _(2,v)+2·d _(3,v))<3·β

The condition for judging whether or not to apply a deblocking filterfor the fifth segment of the boundary which is corresponding to thefifth pixel position is the following:

(2·d _(2,v)+4·d _(3,v))/3<β

Alternatively, the condition for judging whether or not to apply adeblocking filter for the fifth segment of the boundary which iscorresponding to the fifth pixel position is the following:

(2·d _(2,v)+4·d _(3,v))<3·β

The condition for judging whether or not to apply a deblocking filterfor the sixth segment of the boundary which corresponds to the sixthpixel position is the following:

2·d _(3,v)<β

The condition for judging whether or not to apply a deblocking filterfor the seventh segment of the boundary which corresponds to the seventhpixel position is the following:

d _(3,v) +d _(4,v)<β

The condition for judging whether or not to apply a deblocking filterfor the eighths segment of the boundary which corresponds to the eightpixel position is the following:

2·d _(4,v)<β

In the case one of the above conditions is true, the filtering isapplied to the respective individual segment of the vertical boundary.According to the above approach, individual decisions for segments, areperformed by using linear combinations of the values d_(1,v), d_(2,v),d_(3,v) and d_(4,v) (line decision terms). Moreover, the above approachcorresponds to an interpolation of individual decision values obtainedfor segments extending over one pixel position at the boundary. Further,it is understood that the same approach can be applied for judgingwhether or not to apply a deblocking filter at a horizontaledge/boundary.

To summarize, in order to deblock with a high coding efficiency and lowcomputational expense and low memory bandwidth, decision and/or the linedecision terms are calculated not for each individual position (as alsofor JCTVC-C403 and JCTVC-D263). This leads to limited memory bandwidthand limited computational expense. However, individual functions of thecalculated values (line decision terms) are used in order to performindividual and accurate decisions at each position of an edge. A generalexample is shown in FIG. 14 and FIG. 15. A more specific example isshown in FIG. 16 and FIG. 17. As a specific solution, also calculatedvalues of other, e.g. neighboring, segments are used in the function,see FIG. 18 and FIG. 19. It may be beneficial to use a regulardistribution of the positions used to calculate the values, see FIG. 20.Specific a further specific solution, for each segment of an edge of 8edge positions, 4 values are calculated, see FIG. 21-22. For each of theedge positions, individual decisions are performed by the use of linearcombinations of the 4 calculated values. The effect of the invention isto increase of coding efficiency with same low computational expense andsame low memory bandwidth.

In the following, the efficiency of the present invention over prior artis shown as an example. In the HM2.0, one single decision for enablingthe deblocking is performed for an edge segment of eight columns/linesusing two calculated decision values. In contrast to the HM2.0,H.264/MPEG-4 AVC uses eight individual decisions based on eightindividually calculated decision values for each edge segment. Thechange of the decisions to ones similar as in H.264/MPEG-4 AVC canreduce the bit rate at the same quality by 0.2% in average over all testcases. However, the calculation of additional decision values isassociated with additional computational expense. In order to achievethe same average bit rate reduction at a lower additional computationalexpense, a modification of the decisions is invented. The inventionperforms eight individual decisions but needs to calculate only fourdecision values for each edge segment. The same average bit ratereduction of 0.2% is achieved compared to HM2.0 (I-HE: 0.1%, I-LC: 0.1%,RA-HE: 0.2%, RA-LC: 0.2%, LD-HE: 0.3%, LD-LC: 0.3%) with approximatelyno encoder/decoder run time increase in average. For the low delay highefficiency configuration, an average bit rate reduction of 0.7% inachieved for the Class E sequences. An increased subjective quality isnoticeable at the same bit rate.

The current HM 2.0 (see for instance, HM2.0 software:http://hevc.kw.bbc.co.uk/trac/browser/tags/HM-2.0 and T. Wiegand, W.-J.Han, J.-R. Ohm, G. J. Sullivan, High Efficiency Video Coding (HEVC) textspecification Working Draft 1, JCTVC-C403, Guangzou, China, October2010, both is in the following referred to as HM 2.0) applies hybridcoding. In FIG. 23 the generalized block diagram of the hybrid coder isshown. In a first step, the input signal to be coded is predictedblock-wise by either motion compensated prediction or Intra prediction.The resulting prediction error is block-wise transform coded by applyingan approximation of the discrete cosine transform (Integer DCT) followedby a quantization of the coefficients. Due to the block wise motioncompensated prediction and a block wise prediction error coding, socalled blocking artifacts often become visible in the decoded images.These blocking artifacts tend to be annoying for human observers. Inorder to reduce these annoying blocking artifacts, an adaptivedeblocking filter is applied. The deblocked signal is further filteredby the use of an adaptive loop filter before being output and stored forfurther predictions. FIG. 24 illustrates the signal before and after thedeblocking filter for a region of the example test sequence Kimono.

The deblocking of images is performed based on coding units (CU), whichmay have various sizes, e.g. 8×8 samples, 16×16 samples. Vertical andhorizontal edges of prediction and transform blocks are deblocked. Eachedge consists of one or several segments, whereas a segment consists of8 consecutive lines or columns. The segments v_(i) of the vertical edgesare deblocked before the segments h_(i) of the horizontal edges. FIG. 25shows an example coding unit of the size 16×16 samples and the positionsof the corresponding 4 segments v₁, . . . ,v₄ and four segments h₁, . .. ,h₄. The order of deblocking the vertical edges is from top to bottomand from left to right. The order of deblocking the horizontal edges isfrom left to right and from top to bottom. In the following, the sampleson the respective sides of the segments of the edges are denoted as Aand B, see FIG. 26 (from JCT-VC, Test Model under Consideration,JCTVC-B205_draft007, Geneva, Switzerland, 21-28 Jul. 2010). The segmentA corresponds to the left neighboring partition to B for vertical edgesand to the above neighboring partition to B for horizontal edges. Foreach segment of 8 lines/columns, the decisions and filtering operationsare performed as explained in the following section.

In a first step, in the decisions according to the HM2.0, the two valesd₂ and d₅ are calculated by the use of the samples of two lines/columnsas illustrated in FIG. 27:

d ₂ =|p2₂−2·p1₂ +p0₂ |+|q2₂−2·q1₂ +q0₂|

d ₅ =|p2₅−2·p1₅ +p0₅ |+|q2₅−2·q1₅ +q0₅|.

By the use of the two values d₂ and d₅, it is decided by the thresholdoperation

d ₂ +d ₅<β

if all 8 lines/columns of the corresponding segment are filtered or not.In order to perform the decisions, 20 operations are required for eachsegment of 8 lines/columns.

In contrast to the HM2.0, H.264/MPEG-4 AVC applies individual decisions(decisions similar as in H.264/MPEG-4 AVC) for each line/column. Inorder to investigate decisions similar as in H.264/MPEG-4 AVC, anindividual value d_(i) is calculated for each of the 8 lines/columns asillustrated in FIG. 28:

d _(i) =p2_(i)−2·p1_(i) +p0_(i) |+|q2_(i)−2·q1_(i) +q0_(i)| with i=0, .. . ,7.

By the use of the individual values d_(i), it is decided for eachline/column by the threshold operation

2·d _(i)<β

if a line/column of the corresponding segment is filtered or not. Inorder to perform the decisions, 88 operations are required for eachsegment of 8 lines/columns.

In order to perform the decisions for a segment of 8 lines/columns,HM2.0 requires 20 operations. If the decisions are performed similar asin H.264/MPEG-4 AVC, 88 operations are required.

In this embodiment, decisions are proposed which compromise the ones ofHM2.0 and H.264/MPEG-4 AVC with respect to computational expense,measured by number of required operations. Four values d₀, d₂, d₅, andd₇ are calculated for each segment of 8 lines/columns as illustrated inFIG. 29:

d _(i) =|p2_(i)−2·p1_(i) +p0_(i) |+|q2_(i)−2·q1_(i) +q0_(i)| withi=0,2,5,7.

By the use of these values, it is decided for each individualline/column by the threshold operations

2·d _(i)<β for i=0,2,5,7

d ₀ +d ₂<β for i=1

d ₅ +d ₇<β for i=6

(4·d ₂+2·d ₅)<3·β for i=3

(4·d ₅+2·d ₂)<3·β for i=4

if a line/column of the corresponding segment is filtered or not. Inorder to perform the decisions, only 58 operations are required for eachsegment of 8 lines/columns.

Experiments and results are described in the following. The decisionssimilar as in H.264/MPEG-4 AVC, as well as the decisions compromisingHM2.0 and H.264/MPEG-4 AVC, are both integrated into the referencesoftware of HM2.0.

Experiments and results for BD-bit rate and run time ratios aredescribed in the following. Following the common conditions (see forinstance, F. Bossen, Common test conditions and software referenceconfigurations, JCTVC-D500, Daegu, Korea, January, 2011) the performanceof all six test cases is evaluated, which is Intra, Random access, andLow delay, each in high efficiency and low complexity operation mode.For all run time measurements, computers of the same configuration areused.

The BD-rate results as well as the encoder-/decoder run time ratioscompared to the reference HM2.0 are shown in FIG. 30 for the decisionssimilar as in H.264/MPEG-4 AVC and in FIG. 31 for the decisionscompromising HM2.0 and H.264/MPEG-4 AVC. Negative BD-rate numbers show again compared to the reference. Run-time ratios less than 100% showreflect that the run time is lower than the one of the reference. Thefollowing results can be observed for both cases: The bit rate reductionis 0.2% in average of over all test sequences and configurations and0.7% in average for LD-LC, Class E. Approximately no encoder-/decoderrun time increases in average.

A subjective evaluation is described in the following. In CE12, varioustest sequences have been selected for subjective evaluations. For thesetest sequences, the subjective quality of the proposal compared to thereference has been performed with the results shown in the table of FIG.32. For five out of the six test sequences, no difference in subjectivequality is noticeable. For one out of the six test sequences, theproposal is clearly sharper than the reference without increasedblocking. In addition, the proposal shows less color artifacts.

The increase of the sharpness is illustrated in FIG. 33 and FIG. 34. InFIG. 33, a cropped part of a deblocked frame of the test sequence Vidyo3is shown for the case of the reference HM2.0, low delay, highefficiency, QP37. FIG. 34 shows the same cropped part for the case ofthe proposed deblocking.

The reduction of color artifacts is illustrated in FIG. 35, a croppedpart of a deblocked frame of the test sequence Vidyo3 is shown for thecase of the reference HM2.0, low delay, high efficiency, QP37. FIG. 36shows the same cropped part for the case of the proposed deblocking.

In the following the coding efficiency versus the complexity isdescribed. In FIG. 37, the achieved bit rate reduction averaged over alltest cases and test sequences is shown versus the additional number ofrequired operations per edge segment of 8 lines/columns, both comparedto the reference HM2.0. It can be observed that the decisionscompromising H.264/MPEG-4 AVC achieve the same average bit ratereduction of 0.2% compared to the reference but with 44% less operationsthan decisions similar as in H.264/MPEG-4 AVC.

All embodiments of the present invention as described above can becombined.

The processing described in each of embodiments can be simplyimplemented in an independent computer system, by recording, in arecording medium, a program for implementing the configurations of thevideo coding method and the video decoding method described in each ofembodiments. The recording media may be any recording media as long asthe program can be recorded, such as a magnetic disk, an optical disk, amagnetic optical disk, an IC card, and a semiconductor memory.

Hereinafter, the applications to the video coding method and the videodecoding method described in each of embodiments and systems usingthereof will be described.

FIG. 38 illustrates an overall configuration of a content providingsystem ex100 for implementing content distribution services. The areafor providing communication services is divided into cells of desiredsize, and base stations ex106, ex107, ex108, ex109, and ex110 which arefixed wireless stations are placed in each of the cells.

The content providing system ex100 is connected to devices, such as acomputer ex111, a personal digital assistant (PDA) ex112, a cameraex113, a cellular phone ex114 and a game machine ex115, via the Internetex101, an Internet service provider ex102, a telephone network ex104, aswell as the base stations ex106 to ex110, respectively.

However, the configuration of the content providing system ex100 is notlimited to the configuration shown in FIG. 38, and a combination inwhich any of the elements are connected is acceptable. In addition, eachdevice may be directly connected to the telephone network ex104, ratherthan via the base stations ex106 to ex110 which are the fixed wirelessstations. Furthermore, the devices may be interconnected to each othervia a short distance wireless communication and others.

The camera ex113, such as a digital video camera, is capable ofcapturing video. A camera ex116, such as a digital video camera, iscapable of capturing both still images and video.

Furthermore, the cellular phone ex114 may be the one that meets any ofthe standards such as Global System for Mobile Communications (GSM),Code Division Multiple Access (CDMA), Wideband-Code Division MultipleAccess (W-CDMA), Long Term Evolution (LTE), and High Speed Packet Access(HSPA). Alternatively, the cellular phone ex114 may be a PersonalHandyphone System (PHS).

In the content providing system ex100, a streaming server ex103 isconnected to the camera ex113 and others via the telephone network ex104and the base station ex109, which enables distribution of images of alive show and others. In such a distribution, a content (for example,video of a music live show) captured by the user using the camera ex113is coded as described above in each of embodiments, and the codedcontent is transmitted to the streaming server ex103. On the other hand,the streaming server ex103 carries out stream distribution of thetransmitted content data to the clients upon their requests. The clientsinclude the computer ex111, the PDA ex112, the camera ex113, thecellular phone ex114, and the game machine ex115 that are capable ofdecoding the above-mentioned coded data. Each of the devices that havereceived the distributed data decodes and reproduces the coded data.

The captured data may be coded by the camera ex113 or the streamingserver ex103 that transmits the data, or the coding processes may beshared between the camera ex113 and the streaming server ex103.Similarly, the distributed data may be decoded by the clients or thestreaming server ex103, or the decoding processes may be shared betweenthe clients and the streaming server ex103. Furthermore, the data of thestill images and video captured by not only the camera ex113 but alsothe camera ex116 may be transmitted to the streaming server ex103through the computer ex111. The coding processes may be performed by thecamera ex116, the computer ex111, or the streaming server ex103, orshared among them.

Furthermore, the coding and decoding processes may be performed by anLSI ex500 generally included in each of the computer ex111 and thedevices. The LSI ex500 may be configured of a single chip or a pluralityof chips. Software for coding and decoding video may be integrated intosome type of a recording medium (such as a CD-ROM, a flexible disk, anda hard disk) that is readable by the computer ex111 and others, and thecoding and decoding processes may be performed using the software.Furthermore, when the cellular phone ex114 is equipped with a camera,the image data obtained by the camera may be transmitted. The video datais data coded by the LSI ex500 included in the cellular phone ex114.

Furthermore, the streaming server ex103 may be composed of servers andcomputers, and may decentralize data and process the decentralized data,record, or distribute data.

As described above, the clients may receive and reproduce the coded datain the content providing system ex100. In other words, the clients canreceive and decode information transmitted by the user, and reproducethe decoded data in real time in the content providing system ex100, sothat the user who does not have any particular right and equipment canimplement personal broadcasting.

Aside from the example of the content providing system ex100, at leastone of the video coding apparatus and the video decoding apparatusdescribed in each of embodiments may be implemented in a digitalbroadcasting system ex200 illustrated in FIG. 39. More specifically, abroadcast station ex201 communicates or transmits, via radio waves to abroadcast satellite ex202, multiplexed data obtained by multiplexingaudio data and others onto video data. The video data is data coded bythe video coding method described in each of embodiments. Upon receiptof the multiplexed data, the broadcast satellite ex202 transmits radiowaves for broadcasting. Then, a home-use antenna ex204 with a satellitebroadcast reception function receives the radio waves.

Next, a device such as a television (receiver) ex300 and a set top box(STB) ex217 decodes the received multiplexed data, and reproduces thedecoded data.

Furthermore, a reader/recorder ex218 (i) reads and decodes themultiplexed data recorded on a recording media ex215, such as a DVD anda BD, or (i) codes video signals in the recording medium ex215, and insome cases, writes data obtained by multiplexing an audio signal on thecoded data. The reader/recorder ex218 can include the video decodingapparatus or the video coding apparatus as shown in each of embodiments.In this case, the reproduced video signals are displayed on the monitorex219, and can be reproduced by another device or system using therecording medium ex215 on which the multiplexed data is recorded. It isalso possible to implement the video decoding apparatus in the set topbox ex217 connected to the cable ex203 for a cable television or to theantenna ex204 for satellite and/or terrestrial broadcasting, so as todisplay the video signals on the monitor ex219 of the television ex300.The video decoding apparatus may be implemented not in the set top boxbut in the television ex300.

FIG. 40 illustrates the television (receiver) ex300 that uses the videocoding method and the video decoding method described in each ofembodiments. The television ex300 includes: a tuner ex301 that obtainsor provides multiplexed data obtained by multiplexing audio data ontovideo data, through the antenna ex204 or the cable ex203, etc. thatreceives a broadcast; a modulation/demodulation unit ex302 thatdemodulates the received multiplexed data or modulates data intomultiplexed data to be supplied outside; and amultiplexing/demultiplexing unit ex303 that demultiplexes the modulatedmultiplexed data into video data and audio data, or multiplexes videodata and audio data coded by a signal processing unit ex306 into data.

The television ex300 further includes: a signal processing unit ex306including an audio signal processing unit ex304 and a video signalprocessing unit ex305 that decode audio data and video data and codeaudio data and video data, respectively; and an output unit ex309including a speaker ex307 that provides the decoded audio signal, and adisplay unit ex308 that displays the decoded video signal, such as adisplay. Furthermore, the television ex300 includes an interface unitex317 including an operation input unit ex312 that receives an input ofa user operation. Furthermore, the television ex300 includes a controlunit ex310 that controls overall each constituent element of thetelevision ex300, and a power supply circuit unit ex311 that suppliespower to each of the elements. Other than the operation input unitex312, the interface unit ex317 may include: a bridge ex313 that isconnected to an external device, such as the reader/recorder ex218; aslot unit ex314 for enabling attachment of the recording medium ex216,such as an SD card; a driver ex315 to be connected to an externalrecording medium, such as a hard disk; and a modem ex316 to be connectedto a telephone network. Here, the recording medium ex216 canelectrically record information using a non-volatile/volatilesemiconductor memory element for storage. The constituent elements ofthe television ex300 are connected to each other through a synchronousbus.

First, the configuration in which the television ex300 decodesmultiplexed data obtained from outside through the antenna ex204 andothers and reproduces the decoded data will be described. In thetelevision ex300, upon a user operation through a remote controllerex220 and others, the multiplexing/demultiplexing unit ex303demultiplexes the multiplexed data demodulated by themodulation/demodulation unit ex302, under control of the control unitex310 including a CPU. Furthermore, the audio signal processing unitex304 decodes the demultiplexed audio data, and the video signalprocessing unit ex305 decodes the demultiplexed video data, using thedecoding method described in each of embodiments, in the televisionex300. The output unit ex309 provides the decoded video signal and audiosignal outside, respectively. When the output unit ex309 provides thevideo signal and the audio signal, the signals may be temporarily storedin buffers ex318 and ex319, and others so that the signals arereproduced in synchronization with each other. Furthermore, thetelevision ex300 may read multiplexed data not through a broadcast andothers but from the recording media ex215 and ex216, such as a magneticdisk, an optical disk, and a SD card. Next, a configuration in which thetelevision ex300 codes an audio signal and a video signal, and transmitsthe data outside or writes the data on a recording medium will bedescribed. In the television ex300, upon a user operation through theremote controller ex220 and others, the audio signal processing unitex304 codes an audio signal, and the video signal processing unit ex305codes a video signal, under control of the control unit ex310 using thecoding method described in each of embodiments. Themultiplexing/demultiplexing unit ex303 multiplexes the coded videosignal and audio signal, and provides the resulting signal outside. Whenthe multiplexing/demultiplexing unit ex303 multiplexes the video signaland the audio signal, the signals may be temporarily stored in thebuffers ex320 and ex321, and others so that the signals are reproducedin synchronization with each other. Here, the buffers ex318, ex319,ex320, and ex321 may be plural as illustrated, or at least one buffermay be shared in the television ex300. Furthermore, data may be storedin a buffer so that the system overflow and underflow may be avoidedbetween the modulation/demodulation unit ex302 and themultiplexing/demultiplexing unit ex303, for example.

Furthermore, the television ex300 may include a configuration forreceiving an AV input from a microphone or a camera other than theconfiguration for obtaining audio and video data from a broadcast or arecording medium, and may code the obtained data. Although thetelevision ex300 can code, multiplex, and provide outside data in thedescription, it may be capable of only receiving, decoding, andproviding outside data but not the coding, multiplexing, and providingoutside data.

Furthermore, when the reader/recorder ex218 reads or writes multiplexeddata from or on a recording medium, one of the television ex300 and thereader/recorder ex218 may decode or code the multiplexed data, and thetelevision ex300 and the reader/recorder ex218 may share the decoding orcoding.

As an example, FIG. 41 illustrates a configuration of an informationreproducing/recording unit ex400 when data is read or written from or onan optical disk. The information reproducing/recording unit ex400includes constituent elements ex401, ex402, ex403, ex404, ex405, ex406,and ex407 to be described hereinafter. The optical head ex401 irradiatesa laser spot in a recording surface of the recording medium ex215 thatis an optical disk to write information, and detects reflected lightfrom the recording surface of the recording medium ex215 to read theinformation. The modulation recording unit ex402 electrically drives asemiconductor laser included in the optical head ex401, and modulatesthe laser light according to recorded data. The reproductiondemodulating unit ex403 amplifies a reproduction signal obtained byelectrically detecting the reflected light from the recording surfaceusing a photo detector included in the optical head ex401, anddemodulates the reproduction signal by separating a signal componentrecorded on the recording medium ex215 to reproduce the necessaryinformation. The buffer ex404 temporarily holds the information to berecorded on the recording medium ex215 and the information reproducedfrom the recording medium ex215. The disk motor ex405 rotates therecording medium ex215. The servo control unit ex406 moves the opticalhead ex401 to a predetermined information track while controlling therotation drive of the disk motor ex405 so as to follow the laser spot.The system control unit ex407 controls overall the informationreproducing/recording unit ex400. The reading and writing processes canbe implemented by the system control unit ex407 using variousinformation stored in the buffer ex404 and generating and adding newinformation as necessary, and by the modulation recording unit ex402,the reproduction demodulating unit ex403, and the servo control unitex406 that record and reproduce information through the optical headex401 while being operated in a coordinated manner. The system controlunit ex407 includes, for example, a microprocessor, and executesprocessing by causing a computer to execute a program for read andwrite.

Although the optical head ex401 irradiates a laser spot in thedescription, it may perform high-density recording using near fieldlight.

FIG. 42 illustrates the recording medium ex215 that is the optical disk.On the recording surface of the recording medium ex215, guide groovesare spirally formed, and an information track ex230 records, in advance,address information indicating an absolute position on the diskaccording to change in a shape of the guide grooves. The addressinformation includes information for determining positions of recordingblocks ex231 that are a unit for recording data. Reproducing theinformation track ex230 and reading the address information in anapparatus that records and reproduces data can lead to determination ofthe positions of the recording blocks. Furthermore, the recording mediumex215 includes a data recording area ex233, an inner circumference areaex232, and an outer circumference area ex234. The data recording areaex233 is an area for use in recording the user data. The innercircumference area ex232 and the outer circumference area ex234 that areinside and outside of the data recording area ex233, respectively arefor specific use except for recording the user data. The informationreproducing/recording unit 400 reads and writes coded audio, coded videodata, or multiplexed data obtained by multiplexing the coded audio andvideo data, from and on the data recording area ex233 of the recordingmedium ex215.

Although an optical disk having a layer, such as a DVD and a BD isdescribed as an example in the description, the optical disk is notlimited to such, and may be an optical disk having a multilayerstructure and capable of being recorded on a part other than thesurface. Furthermore, the optical disk may have a structure formultidimensional recording/reproduction, such as recording ofinformation using light of colors with different wavelengths in the sameportion of the optical disk and for recording information havingdifferent layers from various angles.

Furthermore, a car ex210 having an antenna ex205 can receive data fromthe satellite ex202 and others, and reproduce video on a display devicesuch as a car navigation system ex211 set in the car ex210, in thedigital broadcasting system ex200. Here, a configuration of the carnavigation system ex211 will be a configuration, for example, includinga GPS receiving unit from the configuration illustrated in FIG. 40. Thesame will be true for the configuration of the computer ex111, thecellular phone ex114, and others.

FIG. 43A illustrates the cellular phone ex114 that uses the video codingmethod and the video decoding method described in embodiments. Thecellular phone ex114 includes: an antenna ex350 for transmitting andreceiving radio waves through the base station ex110; a camera unitex365 capable of capturing moving and still images; and a display unitex358 such as a liquid crystal display for displaying the data such asdecoded video captured by the camera unit ex365 or received by theantenna ex350. The cellular phone ex114 further includes: a main bodyunit including an operation key unit ex366; an audio output unit ex357such as a speaker for output of audio; an audio input unit ex356 such asa microphone for input of audio; a memory unit ex367 for storingcaptured video or still pictures, recorded audio, coded or decoded dataof the received video, the still pictures, e-mails, or others; and aslot unit ex364 that is an interface unit for a recording medium thatstores data in the same manner as the memory unit ex367.

Next, an example of a configuration of the cellular phone ex114 will bedescribed with reference to FIG. 43B. In the cellular phone ex114, amain control unit ex360 designed to control overall each unit of themain body including the display unit ex358 as well as the operation keyunit ex366 is connected mutually, via a synchronous bus ex370, to apower supply circuit unit ex361, an operation input control unit ex362,a video signal processing unit ex355, a camera interface unit ex363, aliquid crystal display (LCD) control unit ex359, amodulation/demodulation unit ex352, a multiplexing/demultiplexing unitex353, an audio signal processing unit ex354, the slot unit ex364, andthe memory unit ex367.

When a call-end key or a power key is turned ON by a user's operation,the power supply circuit unit ex361 supplies the respective units withpower from a battery pack so as to activate the cell phone ex114.

In the cellular phone ex114, the audio signal processing unit ex354converts the audio signals collected by the audio input unit ex356 invoice conversation mode into digital audio signals under the control ofthe main control unit ex360 including a CPU, ROM, and RAM. Then, themodulation/demodulation unit ex352 performs spread spectrum processingon the digital audio signals, and the transmitting and receiving unitex351 performs digital-to-analog conversion and frequency conversion onthe data, so as to transmit the resulting data via the antenna ex350.

Also, in the cellular phone ex114, the transmitting and receiving unitex351 amplifies the data received by the antenna ex350 in voiceconversation mode and performs frequency conversion and theanalog-to-digital conversion on the data. Then, themodulation/demodulation unit ex352 performs inverse spread spectrumprocessing on the data, and the audio signal processing unit ex354converts it into analog audio signals, so as to output them via theaudio output unit ex356.

Furthermore, when an e-mail in data communication mode is transmitted,text data of the e-mail inputted by operating the operation key unitex366 and others of the main body is sent out to the main control unitex360 via the operation input control unit ex362. The main control unitex360 causes the modulation/demodulation unit ex352 to perform spreadspectrum processing on the text data, and the transmitting and receivingunit ex351 performs the digital-to-analog conversion and the frequencyconversion on the resulting data to transmit the data to the basestation ex110 via the antenna ex350. When an e-mail is received,processing that is approximately inverse to the processing fortransmitting an e-mail is performed on the received data, and theresulting data is provided to the display unit ex358.

When video, still images, or video and audio in data communication modeis or are transmitted, the video signal processing unit ex355 compressesand codes video signals supplied from the camera unit ex365 using thevideo coding method shown in each of embodiments, and transmits thecoded video data to the multiplexing/demultiplexing unit ex353. Incontrast, during when the camera unit ex365 captures video, stillimages, and others, the audio signal processing unit ex354 codes audiosignals collected by the audio input unit ex356, and transmits the codedaudio data to the multiplexing/demultiplexing unit ex353.

The multiplexing/demultiplexing unit ex353 multiplexes the coded videodata supplied from the video signal processing unit ex355 and the codedaudio data supplied from the audio signal processing unit ex354, using apredetermined method.

Then, the modulation/demodulation unit ex352 performs spread spectrumprocessing on the multiplexed data, and the transmitting and receivingunit ex351 performs digital-to-analog conversion and frequencyconversion on the data so as to transmit the resulting data via theantenna ex350.

When receiving data of a video file which is linked to a Web page andothers in data communication mode or when receiving an e-mail with videoand/or audio attached, in order to decode the multiplexed data receivedvia the antenna ex350, the multiplexing/demultiplexing unit ex353demultiplexes the multiplexed data into a video data bit stream and anaudio data bit stream, and supplies the video signal processing unitex355 with the coded video data and the audio signal processing unitex354 with the coded audio data, through the synchronous bus ex370. Thevideo signal processing unit ex355 decodes the video signal using avideo decoding method corresponding to the coding method shown in eachof embodiments, and then the display unit ex358 displays, for instance,the video and still images included in the video file linked to the Webpage via the LCD control unit ex359. Furthermore, the audio signalprocessing unit ex354 decodes the audio signal, and the audio outputunit ex357 provides the audio.

Furthermore, similarly to the television ex300, a terminal such as thecellular phone ex114 probably have 3 types of implementationconfigurations including not only (i) a transmitting and receivingterminal including both a coding apparatus and a decoding apparatus, butalso (ii) a transmitting terminal including only a coding apparatus and(iii) a receiving terminal including only a decoding apparatus. Althoughthe digital broadcasting system ex200 receives and transmits themultiplexed data obtained by multiplexing audio data onto video data inthe description, the multiplexed data may be data obtained bymultiplexing not audio data but character data related to video ontovideo data, and may be not multiplexed data but video data itself.

As such, the video coding method and the video decoding method in eachof embodiments can be used in any of the devices and systems described.Thus, the advantages described in each of embodiments can be obtained.

Furthermore, the present invention is not limited to embodiments, andvarious modifications and revisions are possible without departing fromthe scope of the present invention.

Video data can be generated by switching, as necessary, between (i) thevideo coding method or the video coding apparatus shown in each ofembodiments and (ii) a video coding method or a video coding apparatusin conformity with a different standard, such as MPEG-2, H.264/AVC, andVC-1.

Here, when a plurality of video data that conforms to the differentstandards is generated and is then decoded, the decoding methods need tobe selected to conform to the different standards. However, since towhich standard each of the plurality of the video data to be decodedconform cannot be detected, there is a problem that an appropriatedecoding method cannot be selected.

In order to solve the problem, multiplexed data obtained by multiplexingaudio data and others onto video data has a structure includingidentification information indicating to which standard the video dataconforms. The specific structure of the multiplexed data including thevideo data generated in the video coding method and by the video codingapparatus shown in each of embodiments will be hereinafter described.The multiplexed data is a digital stream in the MPEG2-Transport Streamformat.

FIG. 44 illustrates a structure of the multiplexed data. As illustratedin FIG. 44, the multiplexed data can be obtained by multiplexing atleast one of a video stream, an audio stream, a presentation graphicsstream (PG), and an interactive graphics stream. The video streamrepresents primary video and secondary video of a movie, the audiostream (IG) represents a primary audio part and a secondary audio partto be mixed with the primary audio part, and the presentation graphicsstream represents subtitles of the movie. Here, the primary video isnormal video to be displayed on a screen, and the secondary video isvideo to be displayed on a smaller window in the primary video.Furthermore, the interactive graphics stream represents an interactivescreen to be generated by arranging the GUI components on a screen. Thevideo stream is coded in the video coding method or by the video codingapparatus shown in each of embodiments, or in a video coding method orby a video coding apparatus in conformity with a conventional standard,such as MPEG-2, H.264/AVC, and VC-1. The audio stream is coded inaccordance with a standard, such as Dolby-AC-3, Dolby Digital Plus, MLP,DTS, DTS-HD, and linear PCM.

Each stream included in the multiplexed data is identified by PID. Forexample, 0x1011 is allocated to the video stream to be used for video ofa movie, 0x1100 to 0x111F are allocated to the audio streams, 0x1200 to0x121F are allocated to the presentation graphics streams, 0x1400 to0x141F are allocated to the interactive graphics streams, 0x1 B00 to0x1B1F are allocated to the video streams to be used for secondary videoof the movie, and 0x1A00 to 0x1A1F are allocated to the audio streams tobe used for the secondary video to be mixed with the primary audio.

FIG. 45 schematically illustrates how data is multiplexed. First, avideo stream ex235 composed of video frames and an audio stream ex238composed of audio frames are transformed into a stream of PES packetsex236 and a stream of PES packets ex239, and further into TS packetsex237 and TS packets ex240, respectively. Similarly, data of apresentation graphics stream ex241 and data of an interactive graphicsstream ex244 are transformed into a stream of PES packets ex242 and astream of PES packets ex245, and further into TS packets ex243 and TSpackets ex246, respectively. These TS packets are multiplexed into astream to obtain multiplexed data ex247.

FIG. 46 illustrates how a video stream is stored in a stream of PESpackets in more detail. The first bar in FIG. 20 shows a video framestream in a video stream. The second bar shows the stream of PESpackets. As indicated by arrows denoted as yy1, yy2, yy3, and yy4 inFIG. 20, the video stream is divided into pictures as I pictures, Bpictures, and P pictures each of which is a video presentation unit, andthe pictures are stored in a payload of each of the PES packets. Each ofthe PES packets has a PES header, and the PES header stores aPresentation Time-Stamp (PTS) indicating a display time of the picture,and a Decoding Time-Stamp (DTS) indicating a decoding time of thepicture.

FIG. 47 illustrates a format of TS packets to be finally written on themultiplexed data. Each of the TS packets is a 188-byte fixed lengthpacket including a 4-byte TS header having information, such as a PIDfor identifying a stream and a 184-byte TS payload for storing data. ThePES packets are divided, and stored in the TS payloads, respectively.When a BD ROM is used, each of the TS packets is given a 4-byteTP_Extra_Header, thus resulting in 192-byte source packets. The sourcepackets are written on the multiplexed data. The TP_Extra_Header storesinformation such as an Arrival_Time_Stamp (ATS). The ATS shows atransfer start time at which each of the TS packets is to be transferredto a PID filter. The source packets are arranged in the multiplexed dataas shown at the bottom of FIG. 47. The numbers incrementing from thehead of the multiplexed data are called source packet numbers (SPNs).

Each of the TS packets included in the multiplexed data includes notonly streams of audio, video, subtitles and others, but also a ProgramAssociation Table (PAT), a Program Map Table (PMT), and a Program ClockReference (PCR). The PAT shows what a PID in a PMT used in themultiplexed data indicates, and a PID of the PAT itself is registered aszero. The PMT stores PIDs of the streams of video, audio, subtitles andothers included in the multiplexed data, and attribute information ofthe streams corresponding to the PIDs. The PMT also has variousdescriptors relating to the multiplexed data. The descriptors haveinformation such as copy control information showing whether copying ofthe multiplexed data is permitted or not. The PCR stores STC timeinformation corresponding to an ATS showing when the PCR packet istransferred to a decoder, in order to achieve synchronization between anArrival Time Clock (ATC) that is a time axis of ATSs, and an System TimeClock (STC) that is a time axis of PTSs and DTSs.

FIG. 48 illustrates the data structure of the PMT in detail. A PMTheader is disposed at the top of the PMT. The PMT header describes thelength of data included in the PMT and others. A plurality ofdescriptors relating to the multiplexed data is disposed after the PMTheader. Information such as the copy control information is described inthe descriptors. After the descriptors, a plurality of pieces of streaminformation relating to the streams included in the multiplexed data isdisposed. Each piece of stream information includes stream descriptorseach describing information, such as a stream type for identifying acompression codec of a stream, a stream PID, and stream attributeinformation (such as a frame rate or an aspect ratio). The streamdescriptors are equal in number to the number of streams in themultiplexed data.

When the multiplexed data is recorded on a recording medium and others,it is recorded together with multiplexed data information files.

Each of the multiplexed data information files is management informationof the multiplexed data as shown in FIG. 49. The multiplexed datainformation files are in one to one correspondence with the multiplexeddata, and each of the files includes multiplexed data information,stream attribute information, and an entry map.

As illustrated in FIG. 49, the multiplexed data includes a system rate,a reproduction start time, and a reproduction end time. The system rateindicates the maximum transfer rate at which a system target decoder tobe described later transfers the multiplexed data to a PID filter. Theintervals of the ATSs included in the multiplexed data are set to nothigher than a system rate. The reproduction start time indicates a PTSin a video frame at the head of the multiplexed data. An interval of oneframe is added to a PTS in a video frame at the end of the multiplexeddata, and the PTS is set to the reproduction end time.

As shown in FIG. 50, a piece of attribute information is registered inthe stream attribute information, for each PID of each stream includedin the multiplexed data. Each piece of attribute information hasdifferent information depending on whether the corresponding stream is avideo stream, an audio stream, a presentation graphics stream, or aninteractive graphics stream. Each piece of video stream attributeinformation carries information including what kind of compression codecis used for compressing the video stream, and the resolution, aspectratio and frame rate of the pieces of picture data that is included inthe video stream. Each piece of audio stream attribute informationcarries information including what kind of compression codec is used forcompressing the audio stream, how many channels are included in theaudio stream, which language the audio stream supports, and how high thesampling frequency is. The video stream attribute information and theaudio stream attribute information are used for initialization of adecoder before the player plays back the information.

The multiplexed data to be used is of a stream type included in the PMT.Furthermore, when the multiplexed data is recorded on a recordingmedium, the video stream attribute information included in themultiplexed data information is used. More specifically, the videocoding method or the video coding apparatus described in each ofembodiments includes a step or a unit for allocating unique informationindicating video data generated by the video coding method or the videocoding apparatus in each of embodiments, to the stream type included inthe PMT or the video stream attribute information. With theconfiguration, the video data generated by the video coding method orthe video coding apparatus described in each of embodiments can bedistinguished from video data that conforms to another standard.

Furthermore, FIG. 51 illustrates steps of the video decoding method. InStep exS100, the stream type included in the PMT or the video streamattribute information is obtained from the multiplexed data. Next, inStep exS101, it is determined whether or not the stream type or thevideo stream attribute information indicates that the multiplexed datais generated by the video coding method or the video coding apparatus ineach of embodiments. When it is determined that the stream type or thevideo stream attribute information indicates that the multiplexed datais generated by the video coding method or the video coding apparatus ineach of embodiments, in Step exS102, decoding is performed by the videodecoding method in each of embodiments. Furthermore, when the streamtype or the video stream attribute information indicates conformance tothe conventional standards, such as MPEG-2, H.264/AVC, and VC-1, in StepexS103, decoding is performed by a video decoding method in conformitywith the conventional standards.

As such, allocating a new unique value to the stream type or the videostream attribute information enables determination whether or not thevideo decoding method or the video decoding apparatus that is describedin each of embodiments can perform decoding. Even when multiplexed datathat conforms to a different standard, an appropriate decoding method orapparatus can be selected. Thus, it becomes possible to decodeinformation without any error. Furthermore, the video coding method orapparatus, or the video decoding method or apparatus can be used in thedevices and systems described above.

Each of the video coding method, the video coding apparatus, the videodecoding method, and the video decoding apparatus in each of embodimentsis typically achieved in the form of an integrated circuit or a LargeScale Integrated (LSI) circuit. As an example of the LSI, FIG. 52illustrates a configuration of the LSI ex500 that is made into one chip.The LSI ex500 includes elements ex501, ex502, ex503, ex504, ex505,ex506, ex507, ex508, and ex509 to be described below, and the elementsare connected to each other through a bus ex510. The power supplycircuit unit ex505 is activated by supplying each of the elements withpower when the power supply circuit unit ex505 is turned on.

For example, when coding is performed, the LSI ex500 receives an AVsignal from a microphone ex117, a camera ex113, and others through an AVIO ex509 under control of a control unit ex501 including a CPU ex502, amemory controller ex503, a stream controller ex504, and a drivingfrequency control unit ex512. The received AV signal is temporarilystored in an external memory ex511, such as an SDRAM. Under control ofthe control unit ex501, the stored data is segmented into data portionsaccording to the processing amount and speed to be transmitted to asignal processing unit ex507. Then, the signal processing unit ex507codes an audio signal and/or a video signal. Here, the coding of thevideo signal is the coding described in each of embodiments.Furthermore, the signal processing unit ex507 sometimes multiplexes thecoded audio data and the coded video data, and a stream IO ex506provides the multiplexed data outside. The provided multiplexed data istransmitted to the base station ex107, or written on the recording mediaex215. When data sets are multiplexed, the data should be temporarilystored in the buffer ex508 so that the data sets are synchronized witheach other.

Although the memory ex511 is an element outside the LSI ex500, it may beincluded in the LSI ex500. The buffer ex508 is not limited to onebuffer, but may be composed of buffers. Furthermore, the LSI ex500 maybe made into one chip or a plurality of chips.

Furthermore, although the control unit ex510 includes the CPU ex502, thememory controller ex503, the stream controller ex504, the drivingfrequency control unit ex512, the configuration of the control unitex510 is not limited to such. For example, the signal processing unitex507 may further include a CPU. Inclusion of another CPU in the signalprocessing unit ex507 can improve the processing speed. Furthermore, asanother example, the CPU ex502 may serve as or be a part of the signalprocessing unit ex507, and, for example, may include an audio signalprocessing unit. In such a case, the control unit ex501 includes thesignal processing unit ex507 or the CPU ex502 including a part of thesignal processing unit ex507.

The name used here is LSI, but it may also be called IC, system LSI,super LSI, or ultra LSI depending on the degree of integration.

Moreover, ways to achieve integration are not limited to the LSI, and aspecial circuit or a general purpose processor and so forth can alsoachieve the integration. Field Programmable Gate Array (FPGA) that canbe programmed after manufacturing LSIs or a reconfigurable processorthat allows re-configuration of the connection or configuration of anLSI can be used for the same purpose.

In the future, with advancement in semiconductor technology, a brand-newtechnology may replace LSI. The functional blocks can be integratedusing such a technology. The possibility is that the present inventionis applied to biotechnology.

When video data generated in the video coding method or by the videocoding apparatus described in each of embodiments is decoded, comparedto when video data that conforms to a conventional standard, such asMPEG-2, H.264/AVC, and VC-1 is decoded, the processing amount probablyincreases. Thus, the LSI ex500 needs to be set to a driving frequencyhigher than that of the CPU ex502 to be used when video data inconformity with the conventional standard is decoded. However, when thedriving frequency is set higher, there is a problem that the powerconsumption increases.

In order to solve the problem, the video decoding apparatus, such as thetelevision ex300 and the LSI ex500 is configured to determine to whichstandard the video data conforms, and switch between the drivingfrequencies according to the determined standard. FIG. 53 illustrates aconfiguration ex800. A driving frequency switching unit ex803 sets adriving frequency to a higher driving frequency when video data isgenerated by the video coding method or the video coding apparatusdescribed in each of embodiments. Then, the driving frequency switchingunit ex803 instructs a decoding processing unit ex801 that executes thevideo decoding method described in each of embodiments to decode thevideo data. When the video data conforms to the conventional standard,the driving frequency switching unit ex803 sets a driving frequency to alower driving frequency than that of the video data generated by thevideo coding method or the video coding apparatus described in each ofembodiments. Then, the driving frequency switching unit ex803 instructsthe decoding processing unit ex802 that conforms to the conventionalstandard to decode the video data.

More specifically, the driving frequency switching unit ex803 includesthe CPU ex502 and the driving frequency control unit ex512 in FIG. 26.Here, each of the decoding processing unit ex801 that executes the videodecoding method described in each of embodiments and the decodingprocessing unit ex802 that conforms to the conventional standardcorresponds to the signal processing unit ex507 in FIG. 50. The CPUex502 determines to which standard the video data conforms. Then, thedriving frequency control unit ex512 determines a driving frequencybased on a signal from the CPU ex502. Furthermore, the signal processingunit ex507 decodes the video data based on the signal from the CPUex502. For example, the identification information described is probablyused for identifying the video data. The identification information isnot limited to the one described above but may be any information aslong as the information indicates to which standard the video dataconforms. For example, when which standard video data conforms to can bedetermined based on an external signal for determining that the videodata is used for a television or a disk, etc., the determination may bemade based on such an external signal. Furthermore, the CPU ex502selects a driving frequency based on, for example, a look-up table inwhich the standards of the video data are associated with the drivingfrequencies as shown in FIG. 55. The driving frequency can be selectedby storing the look-up table in the buffer ex508 and in an internalmemory of an LSI, and with reference to the look-up table by the CPUex502.

FIG. 54 illustrates steps for executing a method. First, in Step exS200,the signal processing unit ex507 obtains identification information fromthe multiplexed data. Next, in Step exS201, the CPU ex502 determineswhether or not the video data is generated by the coding method and thecoding apparatus described in each of embodiments, based on theidentification information. When the video data is generated by thevideo coding method and the video coding apparatus described in each ofembodiments, in Step exS202, the CPU ex502 transmits a signal forsetting the driving frequency to a higher driving frequency to thedriving frequency control unit ex512. Then, the driving frequencycontrol unit ex512 sets the driving frequency to the higher drivingfrequency. On the other hand, when the identification informationindicates that the video data conforms to the conventional standard,such as MPEG-2, H.264/AVC, and VC-1, in Step exS203, the CPU ex502transmits a signal for setting the driving frequency to a lower drivingfrequency to the driving frequency control unit ex512. Then, the drivingfrequency control unit ex512 sets the driving frequency to the lowerdriving frequency than that in the case where the video data isgenerated by the video coding method and the video coding apparatusdescribed in each of embodiment.

Furthermore, along with the switching of the driving frequencies, thepower conservation effect can be improved by changing the voltage to beapplied to the LSI ex500 or an apparatus including the LSI ex500. Forexample, when the driving frequency is set lower, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set to a voltage lower than that in the case where the drivingfrequency is set higher.

Furthermore, when the processing amount for decoding is larger, thedriving frequency may be set higher, and when the processing amount fordecoding is smaller, the driving frequency may be set lower as themethod for setting the driving frequency. Thus, the setting method isnot limited to the ones described above. For example, when theprocessing amount for decoding video data in conformity with H.264/AVCis larger than the processing amount for decoding video data generatedby the video coding method and the video coding apparatus described ineach of embodiments, the driving frequency is probably set in reverseorder to the setting described above.

Furthermore, the method for setting the driving frequency is not limitedto the method for setting the driving frequency lower. For example, whenthe identification information indicates that the video data isgenerated by the video coding method and the video coding apparatusdescribed in each of embodiments, the voltage to be applied to the LSIex500 or the apparatus including the LSI ex500 is probably set higher.When the identification information indicates that the video dataconforms to the conventional standard, such as MPEG-2, H.264/AVC, andVC-1, the voltage to be applied to the LSI ex500 or the apparatusincluding the LSI ex500 is probably set lower. As another example, whenthe identification information indicates that the video data isgenerated by the video coding method and the video coding apparatusdescribed in each of embodiments, the driving of the CPU ex502 does notprobably have to be suspended. When the identification informationindicates that the video data conforms to the conventional standard,such as MPEG-2, H.264/AVC, and VC-1, the driving of the CPU ex502 isprobably suspended at a given time because the CPU ex502 has extraprocessing capacity. Even when the identification information indicatesthat the video data is generated by the video coding method and thevideo coding apparatus described in each of embodiments, in the casewhere the CPU ex502 has extra processing capacity, the driving of theCPU ex502 is probably suspended at a given time. In such a case, thesuspending time is probably set shorter than that in the case where whenthe identification information indicates that the video data conforms tothe conventional standard, such as MPEG-2, H.264/AVC, and VC-1.

Accordingly, the power conservation effect can be improved by switchingbetween the driving frequencies in accordance with the standard to whichthe video data conforms. Furthermore, when the LSI ex500 or theapparatus including the LSI ex500 is driven using a battery, the batterylife can be extended with the power conservation effect.

There are cases where a plurality of video data that conforms todifferent standards, is provided to the devices and systems, such as atelevision and a mobile phone. In order to enable decoding the pluralityof video data that conforms to the different standards, the signalprocessing unit ex507 of the LSI ex500 needs to conform to the differentstandards. However, the problems of increase in the scale of the circuitof the LSI ex500 and increase in the cost arise with the individual useof the signal processing units ex507 that conform to the respectivestandards.

In order to solve the problem, what is conceived is a configuration inwhich the decoding processing unit for implementing the video decodingmethod described in each of embodiments and the decoding processing unitthat conforms to the conventional standard, such as MPEG-2, H.264/AVC,and VC-1 are partly shared. Ex900 in FIG. 56A shows an example of theconfiguration. For example, the video decoding method described in eachof embodiments and the video decoding method that conforms to H.264/AVChave, partly in common, the details of processing, such as entropycoding, inverse quantization, deblocking filtering, and motioncompensated prediction. The details of processing to be shared mayinclude use of a decoding processing unit ex902 that conforms toH.264/AVC. In contrast, a dedicated decoding processing unit ex901 isprobably used for other processing unique to the present invention.Since the present invention is characterized by application ofdeblocking filtering, for example, the dedicated decoding processingunit ex901 is used for such filtering. Otherwise, the decodingprocessing unit is probably shared for one of the entropy decoding,inverse quantization, spatial or motion compensated prediction, or allof the processing. The decoding processing unit for implementing thevideo decoding method described in each of embodiments may be shared forthe processing to be shared, and a dedicated decoding processing unitmay be used for processing unique to that of H.264/AVC.

Furthermore, ex1000 in FIG. 56B shows another example in that processingis partly shared. This example uses a configuration including adedicated decoding processing unit ex1001 that supports the processingunique to the present invention, a dedicated decoding processing unitex1002 that supports the processing unique to another conventionalstandard, and a decoding processing unit ex1003 that supports processingto be shared between the video decoding method in the present inventionand the conventional video decoding method. Here, the dedicated decodingprocessing units ex1001 and ex1002 are not necessarily specialized forthe processing of the present invention and the processing of theconventional standard, respectively, and may be the ones capable ofimplementing general processing. Furthermore, the configuration can beimplemented by the LSI ex500.

As such, reducing the scale of the circuit of an LSI and reducing thecost are possible by sharing the decoding processing unit for theprocessing to be shared between the video decoding method in the presentinvention and the video decoding method in conformity with theconventional standard.

Most of the examples have been outlined in relation to an H.264/AVCbased video coding system, and the terminology mainly relates to theH.264/AVC terminology. However, this terminology and the description ofthe various embodiments with respect to H.264/AVC based coding is notintended to limit the principles and ideas of the invention to suchsystems. Also the detailed explanations of the encoding and decoding incompliance with the H.264/AVC standard are intended to better understandthe exemplary embodiments described herein and should not be understoodas limiting the invention to the described specific implementations ofprocesses and functions in the video coding. Nevertheless, theimprovements proposed herein may be readily applied in the video codingdescribed. Furthermore the concept of the invention may be also readilyused in the enhancements of H.264/AVC coding and/or HEVC currentlydiscussed by the JCT-VC.

To summarize, the present invention relates to deblocking filtering,which may be advantageously applied for block-wise encoding and decodingof image or video signal. In particular, the present invention relatesto performing an efficient and accurate decision on whether or not toapply deblocking filtering on an image block. The efficient and accuratedecision is achieved by performing individual decisions on whether ornot to apply deblocking filtering for segments of a boundary betweenadjacent image blocks, wherein the individual decision are based onpixels comprised in a subset of the pixel lines that the image blocksare composed of.

1-26. (canceled)
 27. A method for processing an image to providedeblocking filtering, the image being divided into a plurality ofblocks, each block including a plurality of pixel lines perpendicular toa boundary with an adjacent block, the method comprising: dividing oneor more of the plurality of blocks into a plurality of portions fewerthan the plurality of pixel lines, each of the plurality of portionsconsisting of one or more pixel lines; calculating a decision value foreach of the plurality of portions of a block based on values of pixelsin a subset of pixel lines of the block; and judging whether or not toapply a deblocking filter to each of the plurality of portions of theblock based on the respective decision value calculated for the portion,wherein the judging whether or not to apply a deblocking filter to eachportion includes comparing the respective decision value calculated forthe portion with a threshold value.
 28. The method according to claim27, wherein the calculating a decision value for each of the pluralityof portions of a block includes: calculating at least one line decisionterm based on values of pixels in a single pixel line.
 29. The methodaccording to claim 28, wherein the calculating a decision value for eachof the plurality of portions of a block includes: interpolating betweentwo or more lines decision terms, each of the two or more line decisionterms being based values of pixels in a single pixel line.
 30. Themethod according to claim 28, wherein the calculating a decision valuefor each of the plurality of portions of the block further includes:calculating the decision value based on two or more line decision terms,each of two or more line decision terms being based values of pixels ina single pixel line.
 31. The method according to claim 30, wherein thecalculating a decision value based on two or more line decision termsincludes: calculating linear combinations of the two or more linedecision terms.
 32. The method according to claim 30, wherein thecalculating a decision value based on two or more line decision termsincludes: linearly interpolating the two or more line decision terms.33. The method according to claim 27, wherein, in the calculating adecision value for each of the plurality of portions of a block, thesubset of pixels used to calculate the decision value are regularlydistributed in a direction parallel to the boundary.
 34. The methodaccording to claim 27, further comprising: judging which type ofdeblocking filtering is applied at each of the plurality of portions ofa block.
 35. The method according to claim 27, wherein, in thecalculating a decision value for each of the plurality of portions of ablock, the decision value is calculated based on values of pixels in asubset of pixel lines of blocks whose boundary is being processed. 36.The method according to claim 27, wherein, in the calculating a decisionvalue for each of the plurality of portions of a block, the decisionvalue is calculated based on values of pixels in a subset of pixel linesin one or more blocks adjacent to the blocks whose boundary is beingprocessed.
 37. An apparatus for processing an image to providedeblocking filtering, the image being divided into a plurality ofblocks, each block including a plurality of pixel lines perpendicular toa boundary with an block, the apparatus comprising: a processor; and anon-transitory memory having stored thereon executable instructions,which when executed, cause the processor to perform: dividing one ormore of the plurality of blocks into a plurality of portions fewer thanthe plurality of pixel lines, each of the plurality of portionsconsisting of one or more pixel lines; calculating a decision value foreach of the plurality of portions of a block based on values of pixelsin a subset of pixel lines of the block; and judging whether or not toapply a deblocking filter to each of the plurality of portions of theblock based on the respective decision value calculated for the portion,wherein the judging whether or not to apply a deblocking filter to eachportion includes comparing the respective decision value calculated forthe portion with a threshold value.